rsync memory usage, paid feature request

Matthew S. Hallacy poptix at
Wed Jul 6 06:13:23 GMT 2005


The company I work for uses rsync for backups from client computers,
the memory usage is a problem for a lot of them since they're already
busy doing other important things (databases, web serving, etc).

>From the FAQ:

out of memory

The usual reason for "out of memory" when running rsync is that you
are transferring a _very_ large number of files.  The size of the
files doesn't matter, only the total number of files.

As a rule of thumb you should expect rsync to consume about 100 bytes per
file in the file list. This happens because rsync builds a internal
file list structure containing all the vital details of each file. 
rsync needs to hold structure in memory because it is being constantly

I do have a plan for how to rewrite rsync so that it consumes a fixed
(small) amount of memory no matter how many files are transferred, but
I haven't yet found a spare week of coding time to implement it!

Unfortunately there's no indication of who needs a spare week of coding
time, or how much a week would cost. 

Since it's important (to us, probably others) to have a more memory 
friendly rsync, and we're thankful for the work already done, could the 
person responsible for that comment please respond (publicly or privately)
with a dollar figure to make the necessary changes.

This would be for public consumption, we believe in supporting open source

A CC:'d reply would help (I am not subscribed to the list), but I will 
also watch the archives.


Matthew S. Hallacy 

More information about the rsync mailing list