performance problem of using parallel rsync to stage data from 1 source to multiple destination

Wayne Davison wayned at samba.org
Fri Sep 2 16:45:49 GMT 2005


Since it sounds like disk I/O is your limiting factor, you may wish to
look into updating multiple systems using a batch file.  This requires
all the receiving systems to have identical files in the destination
hierarchy.  You would first create a batch file by performing the
synchronization either to one of the destination systems, or even to a
second physical disk on the sending system in an effort to create the
batch file more quickly (if you avoid a copy to the same physical disk*,
it will avoid having both the sending and receiving disk I/O hitting a
single disk).  The batch-writing command would be:

  rsync -av --write-batch=xfer1 /src/ /dest/

Then, you would update all the (remaining) destinations by reading that
batch file:

  rsync -av --read-batch=xfer1 $dest:/tmp/ &

As long as you're using at least 2.6.3, batch mode should work quite
well (older versions used an experimental implementation that is not
recommended).

*One other thing you might try to create the batch file quickly using a
local copy on a machine that doesn't have a second physical disk:  use
the option --only-write-batch instead of --write-batch.  This would use
an extra copy of the destination files somewhere on the sending system,
but not update it right away, which _might_ save some elapsed time in
the creating of the batch file (you'd have to try it and see).  Then, at
the end of the updating of the remote systems, you would use the batch
file to update your local destination mirror.

..wayne..


More information about the rsync mailing list