Data loss (corruption with rsync)

Fri Dec 4 15:19:00 MST 2009

Discussion in enlightened irc channel (OFTC/#ck) gave possible clue
for solving this, although it's just a quess:

0006 @      con || KordeX >4GB file?
0007 +   KordeX || not all of those are
0007 +   KordeX || i mean most of them are <100MB
0007 @      con  | yes but that's the reason it failed
0008 @      con  | rsync uses the rzip algorithm which was 32bit limited as I
          discovered whenI was updating lrzip
0012 +   KordeX || con > it uses rzip even when not using any --compress?
0012 +   KordeX || or does it use it when using append-verify or fuzzy options?
0012 @      con  | I'm not sure but it uses the rzip for the diff function I
          think
0013 +   KordeX || oh that's the reason i was affected
0013 +   KordeX || i think i should re: re: my email with that update
0013 @      con  | actually I'm guessing
0014 +   KordeX || well i add this anyways, give one clue to them
0014 @      con  | dont quote me on that, but I had all sorts of troubles
          dealing with > 2^32 sized files with the rzip code
0014 @      con  | and I had to rewrite most of the rzip algorithm to be 64bit
          smart
0014 @      con  | which unfortunately also slowed it down quite a lot
0015 @      con  | unless you were working on >4GB files that is, where it sped
          it up

Thanks to Con Kolivas.

-Mikko

2009/12/4 kordex - <kordex at gmail.com>:
> Well, as you all professional users know, I had error with these lines
> which i saw cause 'filesystem full':
>> Running
>> navi:/mnt# rsync --recursive --progress --verbose miracle storage
>> started new copy process with disk speed not cached speed.
>
> because they should have been:
> navi:/mnt# rsync --recursive --progress --verbose miracle/ storage/
> with slashes.
>
> Running that caused copying of just 1. single file:
>
> navi:/mnt# rsync --recursive --progress --verbose miracle/ storage/
> sending incremental file list
> storage/video/series/Sliders full/Season 1/Sliders 1x06 Summer of Love.avi
>   430743552 100%   16.54MB/s    0:00:24 (xfer#1, to-check=1621/5752)
>
> sent 431991665 bytes  received 2348 bytes  12895343.67 bytes/sec
> total size is 808342617848  speedup is 1871.19
> navi:/mnt#
>
> and as you see the speed is way too fast it could have gone through
> all the files, so I guess I have got a problem
>
> -Mikko
>
> 2009/12/4 kordex - <kordex at gmail.com>:
>> Hello,
>>
>> I just rsync'd my 1TB partitions formated with jfs filesystem
>> containing ~720GB data. I saw -3G difference between source and
>> destination which made me suspect possible dataloss. Also I had
>> experienced loss of integrity on my previous copies but I did not find
>> the cause until now.
>>
>> As command line I used following:
>> sh-3.2# rsync --verbose --sparse --append-verify --fuzzy --progress
>> --stats --recursive --times --delete /mnt/miracle /mnt/storage
>>
>> sh-3.2# rsync --version
>> rsync  version 3.0.3  protocol version 30
>> Copyright (C) 1996-2008 by Andrew Tridgell, Wayne Davison, and others.
>> Web site: http://rsync.samba.org/
>> Capabilities:
>>    64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints,
>>    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
>>    append, ACLs, xattrs, iconv, symtimes
>>
>> rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
>> are welcome to redistribute it under certain conditions.  See the GNU
>> General Public Licence for details.
>>
>> sh-3.2# uname -a
>> Linux navi 2.6.31.5-bfs304 #9 Wed Dec 2 02:58:47 EET 2009 ppc GNU/Linux
>>
>> I ran
>>
>> navi:/mnt/miracle# foreach i ( "`cat ../miracle.txt`" )
>> foreach? cksum "$i" >> ../miracle.cksums.txt
>> foreach? end
>>
>> miracle.txt has list of files (via find ./ -type f) from original
>> partition. I did same for storage partition (target)
>>
>> The results of :
>> navi:/mnt# diff miracle.cksums.txt storage.cksums.txt | wc -l
>> were (includes some additional lines due diff):
>> 27232
>>
>> The original count of files:
>> navi:/mnt# wc -l miracle.cksums.txt
>> 29266 miracle.cksums.txt
>>
>> every line matches it's counterpart as seen from diff:
>> lines 9-17:
>>
>> 32,34c32,34
>> < 4171132061 137124390 ./music/(2006) the remote viewer [expanded]
>> [flac]/cd1/01. remote viewing 1.flac
>> < 2227991181 48164220 ./music/(2006) the remote viewer [expanded]
>> [flac]/cd1/02. remote viewing 2.flac
>> < 135428409 132580111 ./music/(2006) the remote viewer [expanded]
>> [flac]/cd1/03. remote viewing 3.flac
>> ---
>>> 319542521 137124390 ./music/(2006) the remote viewer [expanded] [flac]/cd1/01. remote viewing 1.flac
>>> 1127514394 48164220 ./music/(2006) the remote viewer [expanded] [flac]/cd1/02. remote viewing 2.flac
>>> 3162510123 132580111 ./music/(2006) the remote viewer [expanded] [flac]/cd1/03. remote viewing 3.flac
>>
>> lines 3053-3057:
>>
>> 3901c3901
>> < 2907853037 8534775038
>> ./storage/video/watching/anime/Ghost_in_the_Shell_S.A.C._Individual_Eleven_[1080p,BluRay,x264,DTS]_-_THORA/Ghost_in_the_Shell_S.A.C._Individual_Eleven_[1080p,BluRay,x264,DTS]_-_THORA.mkv
>> ---
>>> 2744503124 8534775038 ./storage/video/watching/anime/Ghost_in_the_Shell_S.A.C._Individual_Eleven_[1080p,BluRay,x264,DTS]_-_THORA/Ghost_in_the_Shell_S.A.C._Individual_Eleven_[1080p,BluRay,x264,DTS]_-_THORA.mkv
>>
>>
>> lines 15995-16010:
>>
>> < 2494009766 34132620 ./waffcd/music/Moving Past The
>> Boundaries/Negative Format - 07 Momentum.flac
>> < 3598390059 37450226 ./waffcd/music/Moving Past The
>> Boundaries/Negative Format - 08 Focus.flac
>> < 3539732869 33544123 ./waffcd/music/Moving Past The
>> Boundaries/Negative Format - 09 Spectral Analysis.flac
>> < 1302971449 33702136 ./waffcd/music/Moving Past The
>> Boundaries/Negative Format - 10 Prototype.flac
>> < 911935599 42218545 ./waffcd/music/Moving Past The
>> Boundaries/Negative Format - 11 Heterodox.flac
>> < 2448079640 50651297 ./waffcd/music/Moving Past The
>> Boundaries/Negative Format - 12 Probe v2.4.flac
>> < 3294999483 57501994 ./waffcd/music/Moving Past The
>> Boundaries/Negative Format - 13 Out Of Phase.flac
>> ---
>>> 3567107295 34132620 ./waffcd/music/Moving Past The Boundaries/Negative Format - 07 Momentum.flac
>>> 657686196 37450226 ./waffcd/music/Moving Past The Boundaries/Negative Format - 08 Focus.flac
>>> 2695060787 33544123 ./waffcd/music/Moving Past The Boundaries/Negative Format - 09 Spectral Analysis.flac
>>> 2466439301 33702136 ./waffcd/music/Moving Past The Boundaries/Negative Format - 10 Prototype.flac
>>> 3315348387 42218545 ./waffcd/music/Moving Past The Boundaries/Negative Format - 11 Heterodox.flac
>>> 4227463752 50651297 ./waffcd/music/Moving Past The Boundaries/Negative Format - 12 Probe v2.4.flac
>>> 2982154944 57501994 ./waffcd/music/Moving Past The Boundaries/Negative Format - 13 Out Of Phase.flac
>>
>>
>> As we all know the second field of output from cksum is the size: so
>> the size matches but the content does not.
>>
>> Running
>> navi:/mnt# rsync --recursive --progress --verbose miracle storage
>> started new copy process with disk speed not cached speed.
>>
>> This maybe due fuzzy option but append-verify should (?) confirm that
>> the already existing data matches the appended one (?).
>>
>> Yours,
>>
>> --Mikko Kortelainen
>>
>