sync a lot of files

[Clint Byrum]

ciprian niculescu wrote:

> Hello,
>
> I have a problem with "out of memory", I'm trying to sync around 30mil 
> files and i get error, the sync is on the same host but different 
> directory, this with the 100bytes per file, give me 3G of ram. I have 
> put 12G swap (6 partitions of 2G) and my 3G ram. Last i saw the 
> process rsync get to 2100M of SIZE and 1.8G RSS, don't know exactly at 
> how much it gets till he get the error, but not too far after 2G.
>
> How can i solve the problem, i use 2.6.3 version.
>
> Thanks for advice
>
> Ciprian

You don't say what OS you're running, but I'm going to presume Linux. On 
32bit architectures, on Linux, no one process can access more than 2G of 
RAM. You've made 15G of virtual memory available, but there's just no 
way you can use it with one rsync.

Why are you using rsync for that many files anyway? If I were in your 
position, I'd chop it up into many little chunks (if possible), or if I 
had control over how files were added, I'd have the process that 
adds/changes files keep a list of those changed files.. then just copy 
the ones that changed.

Rsync is an amazing tool.. but it has a window of usefulness, just like 
any other tool. You may have gone beyond that window.