malloc errors? out of memory with many files on HP-UX

Regis_Pei at capgroup.com Regis_Pei at capgroup.com
Tue Nov 11 07:00:20 EST 2003






Have you checked the maxdsiz kernel parameter?  The default is 0x4000000
which is only 64MBs.



|---------+---------------------------------------------->
|         |           "King, Daniel"                     |
|         |           <Daniel.King at Fiserv.com>           |
|         |           Sent by:                           |
|         |           rsync-bounces+rep=capgroup.com at list|
|         |           s.samba.org                        |
|         |                                              |
|         |                                              |
|         |           11/07/2003 09:33 AM                |
|         |                                              |
|---------+---------------------------------------------->
  >------------------------------------------------------------------------------------------------------------------------------|
  |                                                                                                                              |
  |       To:       rsync at lists.samba.org                                                                                        |
  |       cc:       (bcc: Regis Pei/CDS/CG/CAPITAL)                                                                              |
  |       Subject:  malloc errors?  out of memory with many files on HP-UX                                                       |
  >------------------------------------------------------------------------------------------------------------------------------|




Hi, folks.

I've started getting these errors from rsync, and any help would be
appreciated:

>ERROR: out of memory in string_area_new buffer
>rsync error: error allocating core memory buffers (code 22) at util.c(115)
>ERROR: out of memory in string_area_new buffer
>rsync error: error allocating core memory buffers (code 22) at util.c(115)
>ERROR: out of memory in string_area_new buffer
>rsync error: error allocating core memory buffers (code 22) at util.c(115)

The file system transfer in question then fails.  After Googling and
looking
at the man pages, I have surmised that this is likely a malloc/out of
memory
error.  From the man page:

>      RERR_MALLOC 22
>           Error allocating core memory buffers

I'm using version ...

>$ rsync --version
>rsync  version 2.5.6  protocol version 26
>Copyright (C) 1996-2002 by Andrew Tridgell and others
><http://rsync.samba.org/>
>Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles,

>              no IPv6, 32-bit system inums, 64-bit internal inums
>
>rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
>are welcome to redistribute it under certain conditions.  See the GNU
>General Public Licence for details.

HP-UX 11i is the OS flavor, and the file [sub]systems to be transferred
have
many files.  I'll be testing further with more verbosity, but I know that
the files to be transferred would be in the 300,000 count range.  The
actual
quantity of data is fairly small (<20GB).

I also see that ...

"As a rule of thumb you should expect rsync to consume about 100 bytes per
file in the file list."

That would make some 30,000,000 bytes, or ~30MB, which is not much on our
system.  Could there be other issues?  I also see that there is an HP-UX
compile time option (-Ae from
http://samba.org/cgi-bin/cvsweb/rsyncweb/faqbody.html?rev=1.4).  I suspect
that this applies to the HP ANSI compiler, not gcc as I am using, but I do
not know if this is the cause.

I see some memory issues in the TODO list with my distribution, but I'm not
familiar with the fix(es).  It perplexes me that the site above references
80MB transfers - over twice the size of my current issue ... oh wait, my
test is back (with double-v) ...

receiving file list ...
expand file_list to 4000 bytes, did move
expand file_list to 8000 bytes, did move
expand file_list to 8000 bytes, did move
expand file_list to 16000 bytes, did move
expand file_list to 16000 bytes, did move
expand file_list to 32000 bytes, did move
expand file_list to 32000 bytes, did move
expand file_list to 64000 bytes, did move
expand file_list to 64000 bytes, did move
expand file_list to 128000 bytes, did move
expand file_list to 128000 bytes, did move
expand file_list to 256000 bytes, did move
expand file_list to 256000 bytes, did move
expand file_list to 512000 bytes, did move
expand file_list to 512000 bytes, did move
expand file_list to 1024000 bytes, did move
expand file_list to 1024000 bytes, did move
expand file_list to 2048000 bytes, did move
expand file_list to 2048000 bytes, did move
expand file_list to 4096000 bytes, did move
expand file_list to 4096000 bytes, did move
expand file_list to 8192000 bytes, did move
expand file_list to 8192000 bytes, did move
expand file_list to 16384000 bytes, did move
expand file_list to 16384000 bytes, did move
expand file_list to 32768000 bytes, did move
expand file_list to 32768000 bytes, did move
ERROR: out of memory in receive_file_entry
rsync error: error allocating core memory buffers (code 22) at util.c(115)

This looks like one of those pesky sizeof-type issues ... Would a 64-bit
compile help?  Are there other configure/compile options which should help?
Google and the list archives do not return anything on this issue
specifically.

Any direction would be appreciated, even where to dig or specific cvs
information ... thanks in advance!

A. Daniel King, System Analyst
HP-UX, Linux, Solaris
:wq
--
To unsubscribe or change options:
http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html






More information about the rsync mailing list