Out of memory in make_file

[Kelly Kane]

Hello,

    I did some searching through the rsync archives at mail-archive.com 
and didn't find anything less than 2 years old. I am hoping there have 
been some developments since then, or that I can look towards debian for 
clues to this problem.

    I am trying to rsync some pretty large directories. These 
directories have about 100-200 users in them. Sometimes we get this on a 
single users home directory. Normally this works pretty well, but on 
occasion we get repeatable errors:

# rsync -av --numeric-ids
...snip...
ERROR: out of memory in make_file [sender]
rsync: writefd_unbuffered failed to write 4 bytes [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (1960976 bytes received so far) 
[sender]
rsync error: error allocating core memory buffers (code 22) at io.c(548) 
[sender=3.0.0cvs]

We're running this on a system which is in live use a file server. It is 
copying from an NFS mount to an AOE block device. The current output is 
it iterating through a Maildir. There is plenty of free memory available 
on the server as far as I can tell:

bottom:~# free
             total       used       free     shared    buffers     cached
Mem:       3090232    3068536      21696          0        164    2289396
-/+ buffers/cache:     778976    2311256
Swap:            0          0          0

No limits:

bottom:~# ulimit
unlimited

Linux bottom 2.6.17.14-c1 #1 SMP Mon Nov 6 12:35:31 EST 2006 x86_64 
GNU/Linux

It is Debian testing/unstable. We can reproduce this problem on the 
packaged debian version (2.6.7) and the current CVS checkout 
(2007-04-02, around 11am Pacific time.) There is another current rsync 
process running on the server using 171m VIRT, 164m RES, and 728 SHR. It 
is the sender and receiver.

I was wondering if anyone could shed some light on this? Is it still an 
issue of "break it up into smaller hunks?" Any insight is appreciated. I 
can provide any debugging information you need on the system, tuning, 
kernel build information, etc, that you may need.

Thank you in advance,
Kelly