Rsync 2.4.6 and Hammerd CPU's

[Dave Dykstra]

Are you running multiple rsyncs in parallel?  I don't see how a single
rsync can use up 4 CPUs when rsync is only 1 or 2 processes depending on
whether it is sending or receiving, respectively.  If you are running
multiple rsyncs, cut down on the number you're doing in parallel.

Rsync does use up a bit of memory for every file it visits in a single
run.  A redesign to avoid that has long been under discusion but it
hasn't been implemented.

- Dave Dykstra

On Mon, Jun 17, 2002 at 09:24:35AM -0700, Sandy Ganz wrote:
> Hello -
> 
> I'm running Redhat 7.1 Rsync 2.4.6 and am having some problems with the
> CPU's getting swammped. What we have is basically an 3 boxes, 1 machine (the
> factory) is a where applications store the images that eventually are used
> for the site, then the other 2 are the webservers that the have copy of the
> images. The person who set the machines up used rsync to move any new images
> from the 'factory box' to both of the webserver boxes. The amount of images
> are huge on all boxes (close to a million). Typically a few thousands images
> is all that rsync must move. The directory structure where the images live
> are broken down into a shallow and somewhat wide tree with a few thousand
> images in each directory
> 
> The problem...
> 
> When rsync seems to to build the list of files, seems OK, uses about 20meg
> of ram (not swaping either). Then  when the files start to be moved (all
> around 2-4k) the cpu's (4x 500 xeon) all run 100% and kill any Http trafic
> that need also to be running on the box. This would be ok if on the Source
> factory box, but not on the web servers. The interesting thing is that is
> does not matter if I run Rsync on the Factory or the Webservers it only ever
> pegs the CPU's on the web servers.
> 
> I tried the throttle on bandwidth [at 256], was ng, as that might be for
> file transfers of a larger size. This is lots of small .jpg images.
> 
> Any ideas on how to keep rsync from using all the cpu on the webservers? I
> got tossed this problem as I was a warm body, but am not a linux master so
> might be I got stuck with some bad use of rsync.
> 
> Also it seems odd that both the source and dest machings use a huge amount
> of ram, I would expect only one to need to do the file compair. Does this
> sound wrong?
> 
> Thanks for any help you can offer
> 
> Sandy
> 
> 
> 
> -- 
> To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html