rsync freezes when copying several million files

[Tomasz Chmielewski]

initiators at free.fr wrote:
> Hello
> 
> I have a similar problem. I want to rsync a huge number of files and I'm
> running out of memory.

You might give the machine more swap.
As it appeared, it wasn't a freeze - it just took a lot of time to 
generate the list. When I came back in the morning, the files were 
transferred.
As I remember, the first copy (many many files) took about 14 hours to 
complete; the second time could be even slower, as rsync had to compare 
two directories, each with over 7 million files, full of hardlinks.

Over 1 GB was on swap, and swap is not the fastest raplacement for RAM...


> I'm using Dirvish to backup my servers. For those who haven't heard
> about it it's sort of "front end" for rsync making a copy of the files
> for the first backup, and then copying modified files and hardlinking
> unmodified files every day.
> I have 600 000 files, and I'm keeping 120 backups on disk. Files are not
> changing too much, I have something like 120 GB of data and 180 GB of
> backup.
> That is roughly 600 000 files with 100 hardlinks on each, so 60 000 000
> files.

Thats nearly 10 times as much as I had :)


>>From what I've been reading in the FAQ and the ML rsync will use 100
> bytes per files (6 GB), and I'll probably get a x3 "bonus" (18 GB) for
> using the --delete and -H options.
> 
> How could I get these files rsynced ?

Perhaps investing in RAM would be good idea.


> By the way what is the memory usage on the receiver side ?
> For the first rsync there is no files receiver side, so it's no problem.
> But after that first rsync memory usage will be the same on sender and
> receiver side, as they'll both have to create their file list. Am I right ?

You could mount the other side (NFS, iSCSI etc., but that depends on the 
connection you have), so that there would be only one rsync instance 
running?


> My servers are running Debian Sarge i.e. rsync 2.6.4

Upgrade to rsync 2.6.8 on both sides?


-- 
Tomasz Chmielewski
http://wpkg.org