malloc errors? out of memory with many files on HP-UX

jw schultz jw at pegasys.ws
Mon Nov 10 12:41:45 EST 2003


On Fri, Nov 07, 2003 at 11:33:07AM -0600, King, Daniel wrote:
> Hi, folks.
> 
> I've started getting these errors from rsync, and any help would be
> appreciated:
> 
> >ERROR: out of memory in string_area_new buffer
> >rsync error: error allocating core memory buffers (code 22) at util.c(115)
> >ERROR: out of memory in string_area_new buffer
> >rsync error: error allocating core memory buffers (code 22) at util.c(115)
> >ERROR: out of memory in string_area_new buffer
> >rsync error: error allocating core memory buffers (code 22) at util.c(115)
> 
> The file system transfer in question then fails.  After Googling and looking
> at the man pages, I have surmised that this is likely a malloc/out of memory
> error.  From the man page:
> 
> >      RERR_MALLOC 22
> >           Error allocating core memory buffers
> 
> I'm using version ...
> 
> >$ rsync --version 
> >rsync  version 2.5.6  protocol version 26
> >Copyright (C) 1996-2002 by Andrew Tridgell and others
> ><http://rsync.samba.org/>
> >Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, 
> >              no IPv6, 32-bit system inums, 64-bit internal inums
> >
> >rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
> >are welcome to redistribute it under certain conditions.  See the GNU
> >General Public Licence for details.
> 
> HP-UX 11i is the OS flavor, and the file [sub]systems to be transferred have
> many files.  I'll be testing further with more verbosity, but I know that
> the files to be transferred would be in the 300,000 count range.  The actual
> quantity of data is fairly small (<20GB).
> 
> I also see that ...
> 
> "As a rule of thumb you should expect rsync to consume about 100 bytes per
> file in the file list."

Thumbs come in all sizes.  The file list itself consumes
approximately 72 bytes (depending on word size and the size
of some of the datatypes used in stat(2)) per file including
directories.  In addition there is the space for the names.

Running "find . -print|wc" is a halfway decent way to
estimate the space requirements.  Line count * 72 +
character count should give you an overestimate.

> That would make some 30,000,000 bytes, or ~30MB, which is not much on our
> system.  Could there be other issues?  I also see that there is an HP-UX
> compile time option (-Ae from
> http://samba.org/cgi-bin/cvsweb/rsyncweb/faqbody.html?rev=1.4).  I suspect
> that this applies to the HP ANSI compiler, not gcc as I am using, but I do
> not know if this is the cause.
> 
> I see some memory issues in the TODO list with my distribution, but I'm not
> familiar with the fix(es).  It perplexes me that the site above references
> 80MB transfers - over twice the size of my current issue ... oh wait, my
> test is back (with double-v) ...
> 
> receiving file list ...
> expand file_list to 4000 bytes, did move
[snip]
> expand file_list to 16384000 bytes, did move
> expand file_list to 16384000 bytes, did move
> expand file_list to 32768000 bytes, did move
> expand file_list to 32768000 bytes, did move
> ERROR: out of memory in receive_file_entry
> rsync error: error allocating core memory buffers (code 22) at util.c(115)

It uses realloc to expand the file_list.  If "did move"
appears that means that the realloc was not able to simply
extend the list but had to allocate new memory to hold it
and then copy from the old to the new locations.  So when it
died on you it was trying to allocate a new 64MB worth of
memory in addition tot he 32MB it already had for the list
(==96MB) plus whatever is needed for holding all the names
and other data structures.

> This looks like one of those pesky sizeof-type issues ... Would a 64-bit
> compile help?  Are there other configure/compile options which should help?

I doubt it.

> Google and the list archives do not return anything on this issue
> specifically.
> 
> Any direction would be appreciated, even where to dig or specific cvs
> information ... thanks in advance!

The only approach is to split the job up.  Hardlinks
permitting.

-- 
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw at pegasys.ws

		Remember Cernan and Schmitt



More information about the rsync mailing list