Data loss (corruption with rsync)

[kordex -]

Testing 3.0G file:

sh-3.2# du -h /home/kordex/Microsoft.Windows.7.ULTIMATE.x64.OEM.DVD-BIE.iso
3.0G    /home/kordex/Microsoft.Windows.7.ULTIMATE.x64.OEM.DVD-BIE.iso

sh-3.2# mount -o remount,ro /home
sh-3.2# fsck.jfs /dev/hdc1
fsck.jfs version 1.1.12, 24-Aug-2007
processing started: 12/5/2009 0.24.4
Using default parameter: -p
The current device is:  /dev/hdc1
Block size in bytes:  4096
Filesystem size in blocks:  78142042
**Phase 0 - Replay Journal Log
Filesystem is clean.
sh-3.2# mount -o remount,rw /home
sh-3.2# cksum /home/kordex/Microsoft.Windows.7.ULTIMATE.x64.OEM.DVD-BIE.iso
175236102 3117350912
/home/kordex/Microsoft.Windows.7.ULTIMATE.x64.OEM.DVD-BIE.iso
sh-3.2# rsync --verbose --sparse --append-verify --fuzzy --progress
--stats /home/kordex/Microsoft.Windows.7.ULTIMATE.x64.OEM.DVD-BIE.iso
/home/kordex/bar/
Microsoft.Windows.7.ULTIMATE.x64.OEM.DVD-BIE.iso
 3117350912 100%    8.19MB/s    0:06:02 (xfer#1, to-check=0/1)

Number of files: 1
Number of files transferred: 1
Total file size: 3117350912 bytes
Total transferred file size: 3117350912 bytes
Literal data: 3117350912 bytes
Matched data: 0 bytes
File list size: 64
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 3117731558
Total bytes received: 31

sent 3117731558 bytes  received 31 bytes  8576978.24 bytes/sec
total size is 3117350912  speedup is 1.00
sh-3.2#
sh-3.2# cksum /home/kordex/bar/Microsoft.Windows.7.ULTIMATE.x64.OEM.DVD-BIE.iso
175236102 3117350912
/home/kordex/bar/Microsoft.Windows.7.ULTIMATE.x64.OEM.DVD-BIE.iso

Seems <4GB file went fine. No need to test with cp(1).

Testing 5.9GB file:

sh-3.2# dd if=/home/kordex/Microsoft.Windows.7.ULTIMATE.x64.OEM.DVD-BIE.iso
of=/home/kordex/bar/Microsoft.Windows.7.ULTIMATE.x64.OEM.DVD-BIE.iso
conv=notrunc oflag=append
sh-3.2# du -h /home/kordex/bar/Microsoft.Windows.7.ULTIMATE.x64.OEM.DVD-BIE.iso
5.9G    /home/kordex/bar/Microsoft.Windows.7.ULTIMATE.x64.OEM.DVD-BIE.iso
sh-3.2# cksum /home/kordex/bar/Microsoft.Windows.7.ULTIMATE.x64.OEM.DVD-BIE.iso
362659186 6234701824
/home/kordex/bar/Microsoft.Windows.7.ULTIMATE.x64.OEM.DVD-BIE.iso
sh-3.2# rsync --verbose --sparse --append-verify --fuzzy --progress
--stats /home/kordex/bar/Microsoft.Windows.7.ULTIMATE.x64.OEM.DVD-BIE.iso
/home/kordex/bar/foo/
Microsoft.Windows.7.ULTIMATE.x64.OEM.DVD-BIE.iso
 6234701824 100%    9.03MB/s    0:10:58 (xfer#1, to-check=0/1)

Number of files: 1
Number of files transferred: 1
Total file size: 6234701824 bytes
Total transferred file size: 6234701824 bytes
Literal data: 6234701824 bytes
Matched data: 0 bytes
File list size: 64
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 6235463006
Total bytes received: 31

sent 6235463006 bytes  received 31 bytes  9454834.02 bytes/sec
total size is 6234701824  speedup is 1.00
sh-3.2#
sh-3.2# cksum /home/kordex/bar/foo/Microsoft.Windows.7.ULTIMATE.x64.OEM.DVD-BIE.iso

 362659186 6234701824
/home/kordex/bar/foo/Microsoft.Windows.7.ULTIMATE.x64.OEM.DVD-BIE.iso


They seem to match too. No need for cp(1).

I will continue tests to my /mnt/miracle and /mnt/storage by tomorrow.





2009/12/5 Matt McCutchen <matt at mattmccutchen.net>:
> On Fri, 2009-12-04 at 18:21 +0200, kordex - wrote:
>> I just rsync'd my 1TB partitions formated with jfs filesystem
>> containing ~720GB data. I saw -3G difference between source and
>> destination which made me suspect possible dataloss. Also I had
>> experienced loss of integrity on my previous copies but I did not find
>> the cause until now.
>>
>> As command line I used following:
>> sh-3.2# rsync --verbose --sparse --append-verify --fuzzy --progress
>> --stats --recursive --times --delete /mnt/miracle /mnt/storage
>
>> As we all know the second field of output from cksum is the size: so
>> the size matches but the content does not.
>
> OK... can you reproduce the problem by copying some of the source files
> to a different place on the destination filesystem, or on a different
> filesystem?  What if you use cp instead of rsync?  Have you run fsck on
> the source and destination filesystems?
>
>> This maybe due fuzzy option but append-verify should (?) confirm that
>> the already existing data matches the appended one (?).
>
> Yes, when a file transfer is performed.  If the destination already has
> a file that passes the quick check with the source file, no file
> transfer is performed.  In that case, consider --ignore-times to disable
> the quick check.
>
> --
> Matt
>
>