out of memory in build_hash_table

[David Bolen]

Eric Echter [eric.echter at mchsi.com] writes:

> I recently installed rsync 2.5.5 on both my rsync server and client being
> used.  I installed the latest version because I was having problems with
> rsync stalling with version 2.4.6 (I read that 2.5.5 was supposed to clear
> this up or at least give more appropriate errors).  I am still having
> problems with rsync stalling even after upgrading to 2.5.5.  It only
stalls
> in the "/home" tree and there are approximately 385,000 (38.5 MB of memory
> give or take if the 100 bytes/file still pertains) files in that tree.

The key growth factor for the file construction is going to be
per-file information that's about 56 bytes (last I looked) per file.
However, the kicker is that the file storage block gets resized
exponentially as you have more files.  So for 385,000 files it'll
actually be a 512,000 file block of about 30MB.  (So yeah, I suppose
an ~50 byte file chunk in memory growing as a power of 2 might average
out close to 100 bytes/file as an estimate :-))

> ERROR: out of memory in build_hash_table
> rsync error: error allocating core memory buffers (code 22) at util.c(232)

Seems like that's just a real out of memory error.  You'll only get that
error if a malloc() call returned NULL.

I presume there's still enough virtual memory available on the server
at the point when this fails?  Could you be running into a process limit 
on virtual memory?  What's a "ulimit -a" show for a server process?  
I think under Linux the default settings are in /etc/security/limits.conf,
maybe by default processes on the server are limited to 32MB of memory 
or something?

-- David

/-----------------------------------------------------------------------\
 \               David Bolen            \   E-mail: db3l at fitlinxx.com  /
  |             FitLinxx, Inc.            \  Phone: (203) 708-5192    |
 /  860 Canal Street, Stamford, CT  06902   \  Fax: (203) 316-5150     \
\-----------------------------------------------------------------------/