Max files/buffer overflow

[Matt McCutchen]

On Tue, 2008-03-25 at 17:35 -0700, Brian D. McGrew wrote:
> I am having an interesting issue where using the command ‘rsync –avP –
> e ssh’ seems to want to bail out on filesystems that have more than
> about 1.2 million files.
> Using that same command on smaller filesystems works like a charm!
> The server is rsync 2.6.8 protocol 29 and the destination is rsync
> 2.6.3 protocol 28.  The server is Fedora Core 4 and the client is Mac
> OS 10.5.2.
> 
> What am I doing wrong, how can I get around this.  I have about four
> million files on a file system that I need to replicate to a disk in
> this Mac!
> 
> Do I just need to upgrade?  Can someone point me to binary packages
> for Fedora Core 4 and Mac OS 10.5 (Intel)???

I assume you mean rsync is running out of memory due to the large number
of files?  You could upgrade to rsync 3.0.0, which has an incremental
recursion mode that avoids holding the entire file list in memory at one
time.  For the Mac, you could try Rob DuToit's binary:

http://rdutoit.home.comcast.net/~rdutoit/dir/lip.zip

Fedora Core 4 is pretty old; I don't know of a binary package of rsync
3.0.0 for it.  You may be able to rebuild a source RPM from a newer
version of Fedora using "rpmbuild --rebuild".  My source RPM is at:

http://mattmccutchen.net/rpm/rsync-3.0.0-1.matt.src.rpm

Or you could just compile and install rsync from the source tarball at:

http://rsync.samba.org/ftp/rsync/rsync-3.0.0.tar.gz

Matt