[PATCH] Reduce memory usage

[Jindřich Makovička]

Hi,

When using rsync to back up the file system on my laptop, containing a
pretty much default linux desktop, I was wondering how rsync uses over
100MB of RAM it allocates.

It turned out that most of the memory is used for the arrays of file_struct
pointers, most of which end up unused - much more than the actual
file_struct entries. In my case, the peak usage was 135MB of pointers, and
just 1.5MB of the file_struct entries themselves.

The problem seems to be that the default file_list allocation parameters
predate the incremental recursion, which allocates a huge number of small
file lists, while AFAICS originally rsync allocated just one large list.

Applying the attached patch, which reduces the default allocation to 32
pointers, and preallocates 32K pointers only for the main file lists in
send_file_list and recv_file_list, reduces the peak memory usage in my case
from 142MB to 12MB.

Regards,
--
Jindřich Makovička
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20210926/368d909c/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Reduce-memory-usage.patch
Type: text/x-patch
Size: 1519 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/rsync/attachments/20210926/368d909c/0001-Reduce-memory-usage.bin>