rsync many small files

[Paul Slootman]

On Tue 23 Sep 2008, Mag Gam wrote:

> At our lab we have storage with many small files. For example a
> directory can contain over 15,000 files and each file averages about
> 75k. I would like to sync this to another filesystem on a different
> server but I am not sure if there is a rsync tuning flag I can use for
> such a intensive job. I am using rsync 3.0.3. Also, I would like to
> use little memory as possible.

You may also want to look at csync2 http://oss.linbit.com/csync2/
for keeping the copies synchronised, it's perhaps better suited than
rsync where you have such a large collection of files where there are
not that many updates going on (i.e. the searching for updates takes
significantly longer than transferring those updates).


Paul Slootman