size-related rsync bugs?

Dave Howorth dhoworth at mrc-lmb.cam.ac.uk
Wed Jun 29 12:40:36 UTC 2016


I use rsync to download a directory from a remote daemon. This has 
worked well for years but has recently started to give problems. The 
rsync transfer is run by a program called dirvish that is run as a cron 
job. The log looks like this:


ACTION: rsync -vrltH --delete --stats -D --numeric-ids 
--exclude-from=/nfs/scop5/data/msd/sifts-mirror/2016-06-28/exclude 
--link-dest=/nfs/scop5/data/msd/sifts-mirror/2016-06-07/tree 
rsync.ebi.ac.uk::pub/databases/msd/sifts/ 
/nfs/scop5/data/msd/sifts-mirror/2016-06-28/tree

receiving incremental file list
csv/pdb_chain_cath_uniprot.csv
csv/pdb_chain_enzyme.csv
csv/pdb_chain_go.csv
broken pipe
RESULTS: warnings = 0, errors = 1

ACTION: rsync -vrltH --delete --stats -D --numeric-ids 
--exclude-from=/nfs/scop5/data/msd/sifts-mirror/2016-06-28/exclude 
--link-dest=/nfs/scop5/data/msd/sifts-mirror/2016-06-07/tree 
rsync.ebi.ac.uk::pub/databases/msd/sifts/ 
/nfs/scop5/data/msd/sifts-mirror/2016-06-28/tree

receiving incremental file list
csv/pdb_chain_go.csv
broken pipe
RESULTS: warnings = 0, errors = 2

ACTION: rsync -vrltH --delete --stats -D --numeric-ids 
--exclude-from=/nfs/scop5/data/msd/sifts-mirror/2016-06-28/exclude 
--link-dest=/nfs/scop5/data/msd/sifts-mirror/2016-06-07/tree 
rsync.ebi.ac.uk::pub/databases/msd/sifts/ 
/nfs/scop5/data/msd/sifts-mirror/2016-06-28/tree

receiving incremental file list
csv/pdb_chain_go.csv
broken pipe
RESULTS: warnings = 0, errors = 3

ACTION: rsync -vrltH --delete --stats -D --numeric-ids 
--exclude-from=/nfs/scop5/data/msd/sifts-mirror/2016-06-28/exclude 
--link-dest=/nfs/scop5/data/msd/sifts-mirror/2016-06-07/tree 
rsync.ebi.ac.uk::pub/databases/msd/sifts/ 
/nfs/scop5/data/msd/sifts-mirror/2016-06-28/tree

receiving incremental file list
csv/pdb_chain_go.csv
broken pipe
RESULTS: warnings = 0, errors = 4

ACTION: rsync -vrltH --delete --stats -D --numeric-ids 
--exclude-from=/nfs/scop5/data/msd/sifts-mirror/2016-06-28/exclude 
--link-dest=/nfs/scop5/data/msd/sifts-mirror/2016-06-07/tree 
rsync.ebi.ac.uk::pub/databases/msd/sifts/ 
/nfs/scop5/data/msd/sifts-mirror/2016-06-28/tree

receiving incremental file list
csv/pdb_chain_go.csv
broken pipe
RESULTS: warnings = 0, errors = 5


It seems to me that the file pdb_chain_go.csv is triggering the problem. 
I notice that in the last transfer that succeeded on 2016-06-07, it was 
1107298708 bytes whilst on 2016-06-22 it had grown a bit to 1111629050 
so I suppose that one possibility is some size-related bug in rsync?

I downloaded the particular file using FTP without any problems and it 
is a text file that appears to be complete without any unusual features.

I was using rsync 3.0.6 but have just installed and used 3.1.2 with the 
same problem. I do not control the remote daemon and have not as yet 
been able to persuade its administrators to send me its logs. The 
protocol version in use is 30 regardless of which version of rsync I use.

I'd be grateful for any ideas that might get my downloads working again.

Thanks, Dave



More information about the rsync mailing list