Problem syncing large dataset

Maarten Thibaut mthibaut at cisco.com
Thu Jan 15 16:33:50 GMT 2009


Hi,

When using rsync-3.0.2 through 3.0.5, I get this error on a large 
dataset syncing from machine-a to machine-b:

$ /bin/rsync -aHSz /local/. machine-b:/local/.
invalid len passed to map_ptr: -1737287498
rsync error: error in file IO (code 11) at fileio.c(188) [sender=3.0.5]

This happens no matter which side initiates the connection, so this 
fails in the same way:

$ /bin/rsync -aHSz machine-a:/local/. /local/.

Using rsyncd, same thing so it's not caused by SSH:

$ /bin/rsync -aHSz rsync://machine-a/local/. /local/.
invalid len passed to map_ptr: -1737287498
rsync error: error in file IO (code 11) at fileio.c(188) [sender=3.0.5]
rsync: connection unexpectedly closed (27172894 bytes received so far) 
[receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) 
[receiver=3.0.5]
rsync: connection unexpectedly closed (390 bytes received so far) 
[generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) 
[generator=3.0.5]

This is about 1TB of data, in about 45 million files. When I chop up my 
rsyncs and run them on smaller parts of the data (like individual 
subdirs) it works fine, so I am assuming the problem is with a lot of data.

Is this a bug, or a problem that I am causing myself somehow? Do I need 
to do some debugging to help you find the problem?

thanks,
maarten


More information about the rsync mailing list