Data loss (corruption with rsync)

[kordex -]

Hello,

I just rsync'd my 1TB partitions formated with jfs filesystem
containing ~720GB data. I saw -3G difference between source and
destination which made me suspect possible dataloss. Also I had
experienced loss of integrity on my previous copies but I did not find
the cause until now.

As command line I used following:
sh-3.2# rsync --verbose --sparse --append-verify --fuzzy --progress
--stats --recursive --times --delete /mnt/miracle /mnt/storage

sh-3.2# rsync --version
rsync  version 3.0.3  protocol version 30
Copyright (C) 1996-2008 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

sh-3.2# uname -a
Linux navi 2.6.31.5-bfs304 #9 Wed Dec 2 02:58:47 EET 2009 ppc GNU/Linux

I ran

navi:/mnt/miracle# foreach i ( "`cat ../miracle.txt`" )
foreach? cksum "$i" >> ../miracle.cksums.txt
foreach? end

miracle.txt has list of files (via find ./ -type f) from original
partition. I did same for storage partition (target)

The results of :
navi:/mnt# diff miracle.cksums.txt storage.cksums.txt | wc -l
were (includes some additional lines due diff):
27232

The original count of files:
navi:/mnt# wc -l miracle.cksums.txt
29266 miracle.cksums.txt

every line matches it's counterpart as seen from diff:
lines 9-17:

32,34c32,34
< 4171132061 137124390 ./music/(2006) the remote viewer [expanded]
[flac]/cd1/01. remote viewing 1.flac
< 2227991181 48164220 ./music/(2006) the remote viewer [expanded]
[flac]/cd1/02. remote viewing 2.flac
< 135428409 132580111 ./music/(2006) the remote viewer [expanded]
[flac]/cd1/03. remote viewing 3.flac
---
> 319542521 137124390 ./music/(2006) the remote viewer [expanded] [flac]/cd1/01. remote viewing 1.flac
> 1127514394 48164220 ./music/(2006) the remote viewer [expanded] [flac]/cd1/02. remote viewing 2.flac
> 3162510123 132580111 ./music/(2006) the remote viewer [expanded] [flac]/cd1/03. remote viewing 3.flac

lines 3053-3057:

3901c3901
< 2907853037 8534775038
./storage/video/watching/anime/Ghost_in_the_Shell_S.A.C._Individual_Eleven_[1080p,BluRay,x264,DTS]_-_THORA/Ghost_in_the_Shell_S.A.C._Individual_Eleven_[1080p,BluRay,x264,DTS]_-_THORA.mkv
---
> 2744503124 8534775038 ./storage/video/watching/anime/Ghost_in_the_Shell_S.A.C._Individual_Eleven_[1080p,BluRay,x264,DTS]_-_THORA/Ghost_in_the_Shell_S.A.C._Individual_Eleven_[1080p,BluRay,x264,DTS]_-_THORA.mkv


lines 15995-16010:

< 2494009766 34132620 ./waffcd/music/Moving Past The
Boundaries/Negative Format - 07 Momentum.flac
< 3598390059 37450226 ./waffcd/music/Moving Past The
Boundaries/Negative Format - 08 Focus.flac
< 3539732869 33544123 ./waffcd/music/Moving Past The
Boundaries/Negative Format - 09 Spectral Analysis.flac
< 1302971449 33702136 ./waffcd/music/Moving Past The
Boundaries/Negative Format - 10 Prototype.flac
< 911935599 42218545 ./waffcd/music/Moving Past The
Boundaries/Negative Format - 11 Heterodox.flac
< 2448079640 50651297 ./waffcd/music/Moving Past The
Boundaries/Negative Format - 12 Probe v2.4.flac
< 3294999483 57501994 ./waffcd/music/Moving Past The
Boundaries/Negative Format - 13 Out Of Phase.flac
---
> 3567107295 34132620 ./waffcd/music/Moving Past The Boundaries/Negative Format - 07 Momentum.flac
> 657686196 37450226 ./waffcd/music/Moving Past The Boundaries/Negative Format - 08 Focus.flac
> 2695060787 33544123 ./waffcd/music/Moving Past The Boundaries/Negative Format - 09 Spectral Analysis.flac
> 2466439301 33702136 ./waffcd/music/Moving Past The Boundaries/Negative Format - 10 Prototype.flac
> 3315348387 42218545 ./waffcd/music/Moving Past The Boundaries/Negative Format - 11 Heterodox.flac
> 4227463752 50651297 ./waffcd/music/Moving Past The Boundaries/Negative Format - 12 Probe v2.4.flac
> 2982154944 57501994 ./waffcd/music/Moving Past The Boundaries/Negative Format - 13 Out Of Phase.flac


As we all know the second field of output from cksum is the size: so
the size matches but the content does not.

Running
navi:/mnt# rsync --recursive --progress --verbose miracle storage
started new copy process with disk speed not cached speed.

This maybe due fuzzy option but append-verify should (?) confirm that
the already existing data matches the appended one (?).

Yours,

--Mikko Kortelainen