Problems with high load

[Paul Slootman]

On Thu 09 Feb 2006, Matthias Blohm wrote:

> We are having problems with a very high load at the time of an rsync is 
> running.
> Sometimes if 2 or 3 Rsyncs are running simultanously on to the same machine 
> the load goes up on that machine to 10 or more. the datas will be synced in 
> about 2 or 3 minutes with about:
> sent 3277383 bytes  received 3719902 bytes  31590.45 bytes/sec
> total size is 391067345  speedup is 55.89
> 
> But the Load is so high that all the Webserveres are stopping its work.

There are a couple of things you could try:

- try to limit the number of rsyncs to 1. You'll have a lot of disk
  thrashing if 3 rsyncs are competing for disk IO.

- Try to reduce the set of files being rsynced, if possible. E.g. run
  rsync multiple times, once for each subdir instead of running it over
  one larger higher directory. (You don't mention how many files are
  involved; output from --stats could help.)

- If files are generally replaced (and not updated in a smallish way),
  try --whole-file, which will reduce IO, as the old data isn't read
  from disk, it's just replaced.

> We have no bandwith throttling because it is important to get its changes 
> fast.

If you know what files are changed, --files-from might be useful.


Paul Slootman