rsync out of memory at 8 MB although ulimit is 512MB

[Matt McCutchen]

On 8/27/07, mjacob at union06.de <mjacob at union06.de> wrote:
> I encountered something amazing. First I thought there is not
> enough memory allowed through ulimit. ulimit is now set to
> (almost) 512MB but rsync still gets out fo memory at 8MB.

> [sender] expand file_list to 131072 bytes, did move
> [sender] expand file_list to 262144 bytes, did move
> [sender] expand file_list to 524288 bytes, did move
> [sender] expand file_list to 1048576 bytes, did move
> [sender] expand file_list to 2097152 bytes, did move
> [sender] expand file_list to 4194304 bytes, did move
> [sender] expand file_list to 8388608 bytes, did move
> ERROR: out of memory in receive_file_entry

Rsync's output is a bit misleading.  The 8 MB is the size of the array
of *pointers* to the file-list entries.  It's very plausible that the
allocated entries themselves could take up 512 MB while the array of
pointers takes up only 8 MB.  You need to either raise the memory
limit even further, break the rsync run up into smaller ones, or use
the incremental recursion mode in rsync 3.0.0 (which keeps only part
of the file list in memory at a time).

Matt