rsync freezes when copying several million files

initiators at free.fr initiators at free.fr
Tue Aug 22 07:37:42 GMT 2006


Hello

I have a similar problem. I want to rsync a huge number of files and I'm
running out of memory.

I'm using Dirvish to backup my servers. For those who haven't heard
about it it's sort of "front end" for rsync making a copy of the files
for the first backup, and then copying modified files and hardlinking
unmodified files every day.
I have 600 000 files, and I'm keeping 120 backups on disk. Files are not
changing too much, I have something like 120 GB of data and 180 GB of
backup.
That is roughly 600 000 files with 100 hardlinks on each, so 60 000 000
files.

>From what I've been reading in the FAQ and the ML rsync will use 100
bytes per files (6 GB), and I'll probably get a x3 "bonus" (18 GB) for
using the --delete and -H options.

How could I get these files rsynced ?

By the way what is the memory usage on the receiver side ?
For the first rsync there is no files receiver side, so it's no problem.
But after that first rsync memory usage will be the same on sender and
receiver side, as they'll both have to create their file list. Am I right ?

My servers are running Debian Sarge i.e. rsync 2.6.4


More information about the rsync mailing list