Problem with checksum failing on large files

Terry Reed twreed at leapwireless.com
Fri Oct 11 15:17:14 EST 2002


I'm having a problem with large files being rsync'd twice because of the
checksum failing.  The rsync appears to complete on the first pass, but then
is done a second time (with second try successful). When some debug code was
added to receiver.c, I saw that the checksum for the remote file & the temp
file do not match on the first try, so (as expected) it repeats the rsync &
the checksums do match after the retry.  

I have reproduced this behavior with various files ranging from 900 MB to
2.7 GB (though smaller files in several MB range work fine on the first
pass).  I just need to rsync a single large file for this scenario.

The client is rsync 2.5.5 on Solaris 9 & the server is rsync 2.5.5 on AIX
4.3.3.  I see the same results with Solaris 9 rsync 2.5.2 client & Solaris 7
rsync 2.5.2 server, as well as Solaris 9 rsync 2.5.2 client with AIX 2.5.5
server.  I also see the same results for rsync in daemon mode & over SSH. I
used gcc to compile rsync in all cases. 

Is there a different checksum mechanism used on the second pass (e.g.,
different length)?  If so, perhaps there is an issue with large files for
what is used by default for the first pass?

Here is the verbose output:

=======================
$ rsync -avvv --stats --progress rsyncuser at 10.200.1.1::rsync/gb-testfile
gb-testfile

opening tcp connection to 10.200.1.1 port 873
Password: 
receiving file list ... 
recv_file_name(gb-testfile)
received 1 names
1 file to consider
recv_file_list done
get_local_name count=1 gb-testfile
generator starting pid=2500 count=1
recv_generator(gb-testfile,0)
recv_files(1) starting
sending sums for 0
generate_files phase=1
recv_files(gb-testfile)
recv mapped gb-testfile of size 432402669
gb-testfile
  915211191 100%    3.61MB/s    0:04:01
got file_sum
renaming .gb-testfile.nDaO4e to gb-testfile
set modtime of gb-testfile to (1034093089) Tue Oct  8 09:04:49 2002 redoing
gb-testfile(0)
recv_generator(gb-testfile,0)
recv_files phase=1
sending sums for 0
generate_files phase=2
recv_files(gb-testfile)
recv mapped gb-testfile of size 915211191
gb-testfile
  915211191 100%   10.23MB/s    0:01:25
got file_sum
renaming .gb-testfile.oDaO4e to gb-testfile
set modtime of gb-testfile to (1034093089) Tue Oct  8 09:04:49 2002 
recv_files finished

Number of files: 1
Number of files transferred: 2
Total file size: 915211191 bytes
Total transferred file size: 1830422382 bytes
Literal data: 482837431 bytes
Matched data: 1347584951 bytes
File list size: 46
Total bytes written: 1275722
Total bytes read: 483225574

wrote 1275722 bytes  read 483225574 bytes  1154949.45 bytes/sec 
total size is 915211191  speedup is 1.89 
_exit_cleanup(code=0, file=main.c, line=925): about to call exit(0)

=======================


Thanks. 

--
Terry




More information about the rsync mailing list