rsync & io

[Paul Haas]

On Thu, 21 Mar 2002, Jie Gao wrote:

> One problem I have using rsync is that if you have a large dataset,
> rsync will use almost all io, leaving almost none for other processes.

Yup, that's particularly annoying when you use rsync for backups where it
just doesn't matter if it takes one hour or twelve hours.

> I wonder if something can be built into rsync to make it more
> io-friendly so that rsync can coexist with other processes?

I don't think we need to change rsync at all.  Use the "--rsh" parameter
to move the data through a program that limits the bandwidth.  If you're
transferring data between systems, then the bandwidth limiting program can
in turn run the real ssh (or even rsh).  For local transfers, it can run
rsync in daemon mode.

If you're running between Linux systems, and you control the kernel, and
you know more than I do, then you can use QOS, to control bandwidth.  Run
a second ssh server on a nonstandard port, call this the slow-ssh port.
Use QOS to limit the bandwidth used by traffic on that port.  This will
let you limit rsync to say 50% of your bandwidth, even when you've got
more than one rsync process running.

I have not yet configured QOS on my kernel, so I don't know the details.

--
Paul Haas paulh at hamjudo.com