Help -- rsync Causing High Load Averages

[Plugger]

Thanks Wayne,

Disk I/O is indeed the problem. This server is serving primarily images
and video content, and when we try to rsync it, the disks cannot handle
the load. That causes the apache processes to take longer. As a result
of that, more apache process are spawned and waiting and it just
snowballs from there.

I will let everyone know if we find a solution other that getting faster
drives.

BTW, scp will also bring the server to its knees.

Thanks
Ron

Wayne Davison wrote:
> On Tue, Mar 28, 2006 at 04:58:42PM -0800, Plugger wrote:
>> When it runs, however, the load averages on the content1 server
>> continue to grow to the 100s, bringing the server to a practical
>> standstill. After a day of tweaking, and killing the content1 server
>> periodically, I have not been able to reduce the loads much.
> 
> You should first determine what lacking resource is causing the high
> load average.  If the CPU is maxed out, the decision to "nice" rsync is
> a good one, but also consider using the --whole-file option (it will use
> less CPU on the sender, and is a good choice for those with a fast
> network connection and too little CPU).  If it is not CPU, it is
> probably disk I/O, which rsync doesn't have a very good way to ration --
> you can try using --bwlimit (which doesn't really help much during the
> filesystem scan) or try the slow-down.diff in the patches dir (which is
> an overly-simple sleep idiom that only affects the filesystem scan), but
> neither one is really the right solution when rsync is using too much
> I/O (and the OS doesn't have a "nice" equivalent for prioritizing
> multiple process's access to I/O).
> 
> ..wayne..
> 
>