Slow rsync Performance

[jw schultz]

On Wed, Jul 30, 2003 at 12:10:49PM -0400, Jose Binoj wrote:
> Hi all,
> I know this subject is extensively discussed on the mail list but I did not 
> get the solution yet..
> 
> I have a server running RedHat Linux 7.2 (2 CPU, 4 Gb RAM) and 6 client 
> machines running RedHat linux 7.1 ( 1 cpu(2Ghz),512 Mb RAM ). I am backing 
> up the client machine every 15 minutes since these machines generates 
> important data. I am using rsync as a cron job on the server to backup 
> these clients over ssh. It worked fine with a 5 Gb total, 100,000 files. 
> But after that the time it takes to sync increased from 3 min to 12 min and 
> now I have 700,000 files and it takes around 18 min.

Break the job up.  The stat info alone (not including pathnames)
of the flist is about 47MB.  Sadly, rsync doesn't scale the
file list very well. 

> The clients are in the same LAN. My problem is during rsync my client 
> machines is very slow and it even locks up. Assuming the number of files 
> remains 700,000 is there any better solution to this problem? Does running 
> as a daemon increase performance on the client machine?
> The file directory structure in the clients machine remains relatevely 
> same..

Except for ssh's (recommended) encryption the additional CPU
load of passing bytes through rsh or ssh as compared with
the daemon should be barely measurable.

In addition to breaking the job up i'd suggest you implement
locking to make sure there is no more than one job running
per client.


-- 
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw at pegasys.ws

		Remember Cernan and Schmitt