Feature Request - Load Throttling

[Paul Haas]

On Tue, 17 Feb 2004, Marc Perkel wrote:

> jw schultz wrote:

> >On Tue, Feb 17, 2004 at 06:47:19PM -0800, Marc Perkel wrote:

> >>Actually - the problem is disk IO. And the disk IO is what makes the
> >>load levels go up.

If the disk IO is correlated with transfer IO, then --bwlimit will solve
your problem.  If there are lots of files in common, then rsync doesn't
actually need to copy them, and the disk IO can climb while the transfer
IO is tiny.  Then you need something else.

> >>The load level is something that's readable can can
> >>be used to have rsync slow itself down. Nice doesn't do the trick. Nice
> >>helps - but even at nice +19 it still slows the system to a crawl when
> >>backing up from one drive to another.

> >>So - if rsync could watch the load levels and pause every now and then
> >>to put a little space between disk access at high load levels it would
> >>make it a lot friendlier to the system. The reason nice doesn't work is
> >>that once the system call is made to access the disk - nice doesn't apply.

> >This is what process and i/o schedulers are for.

There are some experimental i/o scheduler patches for Linux 2.6, but
I don't think there is anything you would want to run in production.

> What happens is that the server is cooking along just fine serving about
> 2 million hits a day. Load level - according to top is running around
> 0.6 to 2.3 or so - and then rsync kicks in doing a backup between the
> drives and even though I'm running at nice +19 the load kicks up to
> around 50 and several services almost stop. That's why I'm asking for
> this feature.

> I'd rather get a response from real develpers than nay sayers telling me
> things that don't work. Its a real issue and it requires changes in
> rsync to make it work correctly.

It's a real issue, and it isn't specific to rsync.  You've got a webserver
that runs well on your hardware, assuming reasonable disk caching and/or
disk I/O rates.  rsync comes along and reads lots of files, thus clearing
your cache, and it walks the directory tree, doing reads spread all over
the disk.  The reads spread over the disk means the disk heads are
spending lots of time seeking back and forth in the famous elevator
algorithm.  Your processes end up spending a lot of time waiting for that
elevator.

I don't think it requires changes to rsync, certainly nothing significant.
I think you want a separate process to implement your policy.  Something
only a little more complex than this untested perl script:

===============cut here================

#!/usr/bin/perl -w
$tooHigh = 4;  # Max acceptable load average
$checkTime = 10; # Seconds between checking load average
$restTime = 60;  # Seconds to pause disk hog process when load average high
@rsyncPids = @ARGV;
while (1) {  # fix this, script should end when the pids exit.
  if ( LoadAvg() > $tooHigh ) {
    PausePids(@rsyncPids);
    sleep(60);
    ResumePids(@rsyncPids);
  }
  sleep(10);
}
sub LoadAvg {
  $upString = `uptime`;
  ($loadAvg) = ($upString =~ m/load average: (\d+\.\d+)/);
  return $loadAvg;
}

sub PausePids {
  $SigStop = 19;
  kill $SigStop, @_;
}
sub ResumePids {
  $SigCont = 18;
  kill $SigCont, @_;
}

===============cut here================

If the load average climbs too high, it pauses rsync, or whatever pids you
asked to pause.

I don't think rsync needs to be changed at all, provided you avoid
anything involving timeouts.

There are certainly situations where rsync would be the important task,
and it would be the other disk hog process that should pause.

It's debatable whether I count as a real developer.
--
Paul Haas paulh at hamjudo.com