Out of memory in make_file

Kelly Kane kelly at hq.newdream.net
Mon Apr 2 20:45:44 GMT 2007


Hello,

    I did some searching through the rsync archives at mail-archive.com 
and didn't find anything less than 2 years old. I am hoping there have 
been some developments since then, or that I can look towards debian for 
clues to this problem.

    I am trying to rsync some pretty large directories. These 
directories have about 100-200 users in them. Sometimes we get this on a 
single users home directory. Normally this works pretty well, but on 
occasion we get repeatable errors:

# rsync -av --numeric-ids
...snip...
ERROR: out of memory in make_file [sender]
rsync: writefd_unbuffered failed to write 4 bytes [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (1960976 bytes received so far) 
[sender]
rsync error: error allocating core memory buffers (code 22) at io.c(548) 
[sender=3.0.0cvs]

We're running this on a system which is in live use a file server. It is 
copying from an NFS mount to an AOE block device. The current output is 
it iterating through a Maildir. There is plenty of free memory available 
on the server as far as I can tell:

bottom:~# free
             total       used       free     shared    buffers     cached
Mem:       3090232    3068536      21696          0        164    2289396
-/+ buffers/cache:     778976    2311256
Swap:            0          0          0

No limits:

bottom:~# ulimit
unlimited

Linux bottom 2.6.17.14-c1 #1 SMP Mon Nov 6 12:35:31 EST 2006 x86_64 
GNU/Linux

It is Debian testing/unstable. We can reproduce this problem on the 
packaged debian version (2.6.7) and the current CVS checkout 
(2007-04-02, around 11am Pacific time.) There is another current rsync 
process running on the server using 171m VIRT, 164m RES, and 728 SHR. It 
is the sender and receiver.

I was wondering if anyone could shed some light on this? Is it still an 
issue of "break it up into smaller hunks?" Any insight is appreciated. I 
can provide any debugging information you need on the system, tuning, 
kernel build information, etc, that you may need.

Thank you in advance,
Kelly


More information about the rsync mailing list