rsync memory usage, paid feature request

Matthew S. Hallacy poptix at poptix.net
Wed Jul 6 06:13:23 GMT 2005


Hello, 

The company I work for uses rsync for backups from client computers,
the memory usage is a problem for a lot of them since they're already
busy doing other important things (databases, web serving, etc).

>From the FAQ:

---
out of memory

The usual reason for "out of memory" when running rsync is that you
are transferring a _very_ large number of files.  The size of the
files doesn't matter, only the total number of files.

As a rule of thumb you should expect rsync to consume about 100 bytes per
file in the file list. This happens because rsync builds a internal
file list structure containing all the vital details of each file. 
rsync needs to hold structure in memory because it is being constantly
traversed.

I do have a plan for how to rewrite rsync so that it consumes a fixed
(small) amount of memory no matter how many files are transferred, but
I haven't yet found a spare week of coding time to implement it!
---

Unfortunately there's no indication of who needs a spare week of coding
time, or how much a week would cost. 

Since it's important (to us, probably others) to have a more memory 
friendly rsync, and we're thankful for the work already done, could the 
person responsible for that comment please respond (publicly or privately)
with a dollar figure to make the necessary changes.

This would be for public consumption, we believe in supporting open source
software.

A CC:'d reply would help (I am not subscribed to the list), but I will 
also watch the archives.

Thanks.

-- 
Matthew S. Hallacy 


More information about the rsync mailing list