Data loss (corruption with rsync)

[kordex -]

Well, as you all professional users know, I had error with these lines
which i saw cause 'filesystem full':
> Running
> navi:/mnt# rsync --recursive --progress --verbose miracle storage
> started new copy process with disk speed not cached speed.

because they should have been:
navi:/mnt# rsync --recursive --progress --verbose miracle/ storage/
with slashes.

Running that caused copying of just 1. single file:

navi:/mnt# rsync --recursive --progress --verbose miracle/ storage/
sending incremental file list
storage/video/series/Sliders full/Season 1/Sliders 1x06 Summer of Love.avi
   430743552 100%   16.54MB/s    0:00:24 (xfer#1, to-check=1621/5752)

sent 431991665 bytes  received 2348 bytes  12895343.67 bytes/sec
total size is 808342617848  speedup is 1871.19
navi:/mnt#

and as you see the speed is way too fast it could have gone through
all the files, so I guess I have got a problem

-Mikko

2009/12/4 kordex - <kordex at gmail.com>:
> Hello,
>
> I just rsync'd my 1TB partitions formated with jfs filesystem
> containing ~720GB data. I saw -3G difference between source and
> destination which made me suspect possible dataloss. Also I had
> experienced loss of integrity on my previous copies but I did not find
> the cause until now.
>
> As command line I used following:
> sh-3.2# rsync --verbose --sparse --append-verify --fuzzy --progress
> --stats --recursive --times --delete /mnt/miracle /mnt/storage
>
> sh-3.2# rsync --version
> rsync  version 3.0.3  protocol version 30
> Copyright (C) 1996-2008 by Andrew Tridgell, Wayne Davison, and others.
> Web site: http://rsync.samba.org/
> Capabilities:
>    64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints,
>    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
>    append, ACLs, xattrs, iconv, symtimes
>
> rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
> are welcome to redistribute it under certain conditions.  See the GNU
> General Public Licence for details.
>
> sh-3.2# uname -a
> Linux navi 2.6.31.5-bfs304 #9 Wed Dec 2 02:58:47 EET 2009 ppc GNU/Linux
>
> I ran
>
> navi:/mnt/miracle# foreach i ( "`cat ../miracle.txt`" )
> foreach? cksum "$i" >> ../miracle.cksums.txt
> foreach? end
>
> miracle.txt has list of files (via find ./ -type f) from original
> partition. I did same for storage partition (target)
>
> The results of :
> navi:/mnt# diff miracle.cksums.txt storage.cksums.txt | wc -l
> were (includes some additional lines due diff):
> 27232
>
> The original count of files:
> navi:/mnt# wc -l miracle.cksums.txt
> 29266 miracle.cksums.txt
>
> every line matches it's counterpart as seen from diff:
> lines 9-17:
>
> 32,34c32,34
> < 4171132061 137124390 ./music/(2006) the remote viewer [expanded]
> [flac]/cd1/01. remote viewing 1.flac
> < 2227991181 48164220 ./music/(2006) the remote viewer [expanded]
> [flac]/cd1/02. remote viewing 2.flac
> < 135428409 132580111 ./music/(2006) the remote viewer [expanded]
> [flac]/cd1/03. remote viewing 3.flac
> ---
>> 319542521 137124390 ./music/(2006) the remote viewer [expanded] [flac]/cd1/01. remote viewing 1.flac
>> 1127514394 48164220 ./music/(2006) the remote viewer [expanded] [flac]/cd1/02. remote viewing 2.flac
>> 3162510123 132580111 ./music/(2006) the remote viewer [expanded] [flac]/cd1/03. remote viewing 3.flac
>
> lines 3053-3057:
>
> 3901c3901
> < 2907853037 8534775038
> ./storage/video/watching/anime/Ghost_in_the_Shell_S.A.C._Individual_Eleven_[1080p,BluRay,x264,DTS]_-_THORA/Ghost_in_the_Shell_S.A.C._Individual_Eleven_[1080p,BluRay,x264,DTS]_-_THORA.mkv
> ---
>> 2744503124 8534775038 ./storage/video/watching/anime/Ghost_in_the_Shell_S.A.C._Individual_Eleven_[1080p,BluRay,x264,DTS]_-_THORA/Ghost_in_the_Shell_S.A.C._Individual_Eleven_[1080p,BluRay,x264,DTS]_-_THORA.mkv
>
>
> lines 15995-16010:
>
> < 2494009766 34132620 ./waffcd/music/Moving Past The
> Boundaries/Negative Format - 07 Momentum.flac
> < 3598390059 37450226 ./waffcd/music/Moving Past The
> Boundaries/Negative Format - 08 Focus.flac
> < 3539732869 33544123 ./waffcd/music/Moving Past The
> Boundaries/Negative Format - 09 Spectral Analysis.flac
> < 1302971449 33702136 ./waffcd/music/Moving Past The
> Boundaries/Negative Format - 10 Prototype.flac
> < 911935599 42218545 ./waffcd/music/Moving Past The
> Boundaries/Negative Format - 11 Heterodox.flac
> < 2448079640 50651297 ./waffcd/music/Moving Past The
> Boundaries/Negative Format - 12 Probe v2.4.flac
> < 3294999483 57501994 ./waffcd/music/Moving Past The
> Boundaries/Negative Format - 13 Out Of Phase.flac
> ---
>> 3567107295 34132620 ./waffcd/music/Moving Past The Boundaries/Negative Format - 07 Momentum.flac
>> 657686196 37450226 ./waffcd/music/Moving Past The Boundaries/Negative Format - 08 Focus.flac
>> 2695060787 33544123 ./waffcd/music/Moving Past The Boundaries/Negative Format - 09 Spectral Analysis.flac
>> 2466439301 33702136 ./waffcd/music/Moving Past The Boundaries/Negative Format - 10 Prototype.flac
>> 3315348387 42218545 ./waffcd/music/Moving Past The Boundaries/Negative Format - 11 Heterodox.flac
>> 4227463752 50651297 ./waffcd/music/Moving Past The Boundaries/Negative Format - 12 Probe v2.4.flac
>> 2982154944 57501994 ./waffcd/music/Moving Past The Boundaries/Negative Format - 13 Out Of Phase.flac
>
>
> As we all know the second field of output from cksum is the size: so
> the size matches but the content does not.
>
> Running
> navi:/mnt# rsync --recursive --progress --verbose miracle storage
> started new copy process with disk speed not cached speed.
>
> This maybe due fuzzy option but append-verify should (?) confirm that
> the already existing data matches the appended one (?).
>
> Yours,
>
> --Mikko Kortelainen
>