Other possible solutions to: rsync memory usage, paid feature request

Matthew S. Hallacy poptix at poptix.net
Thu Jul 7 18:03:47 GMT 2005


> 1) Free: break your rsync's into several executions rather than one huge  
> one.  Do several sub-directory trees, each separately.  If your data  
> files are not organized in such a way that they can easily be divided  
> into a reasonable number of sub-directory trees, consider re-organizing  
> them so that they can be: it will pay off in many other sys-admin  
> benefits as well.  

It's customer data, so we have no control over how it's organized. All
we can depend on is the data being in /home, which is what we rsync.
> 2) Cheap: buy more swap space.  These days random-access magnetic  
> storage is running close to 0.50 USD per gig (e.g. here:  
> http://www.buy.com/retail/product.asp?sku=10360313 is 200GB for $105 in  
> the US, including shipping).  At the stated rate of 100 bytes per file,  
> this is enough storage to add 2 billion files to each rsync that you  
> run, for a price that is less than many programmers want for a week of  
> coding.  If you have much more than 2 billion files in each sub-  
> directory tree, you are probably doing something very wrong. :-)  

The servers already have 2-4GB of ram, with another gig of swap. Yes, 
these servers have a LOT of small files.

> 3) Free: If your problem is not that you are running *out* of memory but  
> rather that rsync is (temporarily) 'stealing' the core (solid-state)  
> memory from the other 'more important' (i.e. requiring quicker response  
> time) processes (causing their data to get swapped out, which might  
> reduce response-time when that data later needs to get swapped back in),  
> you might also consider using the operating system to either lock-down  
> the memory used by your important server programs so that it cannot be  

Yes, the problem is that memory is being stolen from the processes that
the servers exist for to begin with. Your solution isn't really viable
though =)

> 4) Expensive: buy more solid-state memory.  Possibly still cheaper than  
> paying for coding, but at any rate, in my experience, more core is  
> rarely the best solution for lack-of-core problems.  

I agree, which is why we're willing to pay someone with rsync coding fu
to fix it =)

Matthew S. Hallacy 

More information about the rsync mailing list