rsync freezes when copying several million files

[Paul Slootman]

On Tue 22 Aug 2006, initiators at free.fr wrote:
> 
> I'm using Dirvish to backup my servers. For those who haven't heard
> about it it's sort of "front end" for rsync making a copy of the files
> for the first backup, and then copying modified files and hardlinking
> unmodified files every day.
> I have 600 000 files, and I'm keeping 120 backups on disk. Files are not
> changing too much, I have something like 120 GB of data and 180 GB of
> backup.
> That is roughly 600 000 files with 100 hardlinks on each, so 60 000 000
> files.
> 
> >From what I've been reading in the FAQ and the ML rsync will use 100
> bytes per files (6 GB), and I'll probably get a x3 "bonus" (18 GB) for
> using the --delete and -H options.

Only for those files that are actually hardlinked on the source side,
AFAIK. --delete shouldn't make a difference, as you need both lists in
memory anyway (the source and destination lists).

--delete-during may help, if the list is very large, as it means the
list won't have to be processed twice, once for the delete phase and
once for the transfer phase.

Note that on 32-bits systems you won't be able to address more than 3GB
memory, however much swap and RAM you have...


> How could I get these files rsynced ?

The way I do that is to rsync separate logical directories.
E.g. I know that I don't have any files that are hardlinked between /usr
and /var, so it's no problem to rsync those separately from each other,
etc.


> My servers are running Debian Sarge i.e. rsync 2.6.4

Upgrading rsync is highly recommended...


Paul Slootman