out of memory in build_hash_table

[Eric Echter]

> The key growth factor for the file construction is going to be
> per-file information that's about 56 bytes (last I looked) per file.
> However, the kicker is that the file storage block gets resized
> exponentially as you have more files.  So for 385,000 files it'll
> actually be a 512,000 file block of about 30MB.  (So yeah, I suppose
> an ~50 byte file chunk in memory growing as a power of 2 might average
> out close to 100 bytes/file as an estimate :-))
>
> > ERROR: out of memory in build_hash_table
> > rsync error: error allocating core memory buffers (code 22) at
util.c(232)
>
> Seems like that's just a real out of memory error.  You'll only get that
> error if a malloc() call returned NULL.
>
> I presume there's still enough virtual memory available on the server
> at the point when this fails?  Could you be running into a process limit
> on virtual memory?  What's a "ulimit -a" show for a server process?
> I think under Linux the default settings are in /etc/security/limits.conf,
> maybe by default processes on the server are limited to 32MB of memory
> or something?

I too thought it was just a simple memory problem, but I'm not so sure it
is.  The following is a copy of the output for "ulimit -a":

root at host [~]# ulimit -a
core file size (blocks)     1000000
data seg size (kbytes)      unlimited
file size (blocks)          unlimited
max locked memory (kbytes)  unlimited
max memory size (kbytes)    unlimited
open files                  1024
pipe size (512 bytes)       8
stack size (kbytes)         8192
cpu time (seconds)          unlimited
max user processes          8191
virtual memory (kbytes)     unlimited

I also checked the /etc/security/limits.conf file and everything in the file
is commented out.  Are there default limits if there are no actual settings
in this file that may be causing problems?  Your assumption about the memory
limit on processes sounds correct, but I can't find any reasoning for this
from the system settings.  Thanks a bunch for the response.