[PATCH] Reduce memory usage
makovick at gmail.com
Sat Oct 2 13:04:54 UTC 2021
Just note this patch has nothing to do with memory consumption vs
performance. It just avoids allocating memory that was left unused anyway.
On Sat, Oct 2, 2021, 12:51 devzero--- via rsync <rsync at lists.samba.org>
> >In the exchange I argued that proper use of ram as a buffer would have
> cut down backup time to minutes instead of days.
> could you give an example where rsync is slowing things down so much due
> to ram constraints or inefficient ram use?
> please mind that disk bandwith and file copy bandwith is not the same.
> random i/o and seek time is the culprit.
> why should rsync use ram for buffering data it copies, if the linux kernel
> / vm subsystem already does this?
> *Gesendet:* Samstag, 02. Oktober 2021 um 12:07 Uhr
> *Von:* "Rupert Gallagher via rsync" <rsync at lists.samba.org>
> *An:* makovick at gmail.com, rsync at lists.samba.org
> *Betreff:* Re: [PATCH] Reduce memory usage
> If you look at my previous exchange in the list, I raised the need for
> more ram usage via a tool option. In the exchange I argued that proper use
> of ram as a buffer would have cut down backup time to minutes instead of
> days. At the time, my proposal was dismissed by someone saying that rsync
> uses as much ram as it needs. I still feel the need to free rsync from this
> mindless constraint, while also welcoming ram usage optimisations such as
> yours in this patch. How hard can it be to allow rsync to use 1GB of ram
> instead of 100MB? The benefit would be huge. In my case, where a supermicro
> server uses a shared bus to transfer data from two disks, the overhead
> caused by frequent small buffer IO is so high that backup time is still
> huge. And I am using server hardware! PC and laptops are even worse.
> -------- Original Message --------
> On Sep 26, 2021, 13:54, Jindřich Makovička via rsync <
> rsync at lists.samba.org> wrote:
> When using rsync to back up the file system on my laptop, containing a
> pretty much default linux desktop, I was wondering how rsync uses over
> 100MB of RAM it allocates.
> It turned out that most of the memory is used for the arrays of
> file_struct pointers, most of which end up unused - much more than the
> actual file_struct entries. In my case, the peak usage was 135MB of
> pointers, and just 1.5MB of the file_struct entries themselves.
> The problem seems to be that the default file_list allocation parameters
> predate the incremental recursion, which allocates a huge number of small
> file lists, while AFAICS originally rsync allocated just one large list.
> Applying the attached patch, which reduces the default allocation to 32
> pointers, and preallocates 32K pointers only for the main file lists in
> send_file_list and recv_file_list, reduces the peak memory usage in my case
> from 142MB to 12MB.
> Jindřich Makovička
> -- Please use reply-all for most replies to avoid omitting the mailing
> list. To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync Before posting, read:
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the rsync