rsync out of memory at 8 MB although ulimit is 512MB

[mjacob at union06.de]

Thanks, Matt, this is a really good answer to my question.
I already wrote a script recursing into directories only
doing smaller parts because I needed a fast solution.

However this needs a little more work because some
inodes in the tree can be removed or changed so
at the target directory there might be some directories
files left. So far this is no problem to me/us.

Unfortunately new versions are not build for AIX
by IBM and our company rely on them. So this will
last a long time ...

Maybe I can argue them into build it ourself. :-)

Kind regards,
  Marco
--
Marco Jacob, PMP
Prisma GmbH, Germany

>On 8/27/07, mjacob at union06.de <mjacob at union06.de> wrote:
>> I encountered something amazing. First I thought there is not
>> enough memory allowed through ulimit. ulimit is now set to
>> (almost) 512MB but rsync still gets out fo memory at 8MB.
>
>> [sender] expand file_list to 131072 bytes, did move
>> [sender] expand file_list to 262144 bytes, did move
>> [sender] expand file_list to 524288 bytes, did move
>> [sender] expand file_list to 1048576 bytes, did move
>> [sender] expand file_list to 2097152 bytes, did move
>> [sender] expand file_list to 4194304 bytes, did move
>> [sender] expand file_list to 8388608 bytes, did move
>> ERROR: out of memory in receive_file_entry
>
>Rsync's output is a bit misleading.  The 8 MB is the size of the array
>of *pointers* to the file-list entries.  It's very plausible that the
>allocated entries themselves could take up 512 MB while the array of
>pointers takes up only 8 MB.  You need to either raise the memory
>limit even further, break the rsync run up into smaller ones, or use
>the incremental recursion mode in rsync 3.0.0 (which keeps only part
>of the file list in memory at a time).
>
>Matt