rsync many small files

Paul Slootman paul+rsync at wurtel.net
Tue Sep 23 11:25:10 GMT 2008


On Tue 23 Sep 2008, Mag Gam wrote:

> At our lab we have storage with many small files. For example a
> directory can contain over 15,000 files and each file averages about
> 75k. I would like to sync this to another filesystem on a different
> server but I am not sure if there is a rsync tuning flag I can use for
> such a intensive job. I am using rsync 3.0.3. Also, I would like to
> use little memory as possible.

You may also want to look at csync2 http://oss.linbit.com/csync2/
for keeping the copies synchronised, it's perhaps better suited than
rsync where you have such a large collection of files where there are
not that many updates going on (i.e. the searching for updates takes
significantly longer than transferring those updates).


Paul Slootman


More information about the rsync mailing list