Other possible solutions to: rsync memory usage, paid feature request

[Matthew S. Hallacy]

[...]

> 1) Free: break your rsync's into several executions rather than one huge  
> one.  Do several sub-directory trees, each separately.  If your data  
> files are not organized in such a way that they can easily be divided  
> into a reasonable number of sub-directory trees, consider re-organizing  
> them so that they can be: it will pay off in many other sys-admin  
> benefits as well.  

It's customer data, so we have no control over how it's organized. All
we can depend on is the data being in /home, which is what we rsync.
 
> 2) Cheap: buy more swap space.  These days random-access magnetic  
> storage is running close to 0.50 USD per gig (e.g. here:  
> http://www.buy.com/retail/product.asp?sku=10360313 is 200GB for $105 in  
> the US, including shipping).  At the stated rate of 100 bytes per file,  
> this is enough storage to add 2 billion files to each rsync that you  
> run, for a price that is less than many programmers want for a week of  
> coding.  If you have much more than 2 billion files in each sub-  
> directory tree, you are probably doing something very wrong. :-)  

The servers already have 2-4GB of ram, with another gig of swap. Yes, 
these servers have a LOT of small files.

> 3) Free: If your problem is not that you are running *out* of memory but  
> rather that rsync is (temporarily) 'stealing' the core (solid-state)  
> memory from the other 'more important' (i.e. requiring quicker response  
> time) processes (causing their data to get swapped out, which might  
> reduce response-time when that data later needs to get swapped back in),  
> you might also consider using the operating system to either lock-down  
> the memory used by your important server programs so that it cannot be  
[snip]

Yes, the problem is that memory is being stolen from the processes that
the servers exist for to begin with. Your solution isn't really viable
though =)

> 4) Expensive: buy more solid-state memory.  Possibly still cheaper than  
> paying for coding, but at any rate, in my experience, more core is  
> rarely the best solution for lack-of-core problems.  

I agree, which is why we're willing to pay someone with rsync coding fu
to fix it =)

-- 
Matthew S. Hallacy