Rsync download traffic

Max Kipness mkipness at geniant.com
Fri Oct 17 23:45:38 EST 2003


> On Thu, Oct 16, 2003 at 09:38:32PM -0500, Max Kipness wrote:
> > > > I started monitoring lan traffic with RRDTool on a 
> linux box the 
> > > > other day that runs rsync, and I've found what I would 
> consider a 
> > > > strange traffic pattern. This linux box rsync
> > > about 2Gb of
> > > > data to a local samba share that is connected to a Windows 2003
> > > > server. Based on the literal data stat, roughly 20% is 
> > > changed nightly
> > > > and uploaded. However, the strange thing is that the traffic
> > > > monitoring shows that there is 3mb/s saturation both ways, 
> > > upload and
> > > > download (somehow the server only had a 10mb ethernet adapter).
> > > >  
> > > > This behavior could not possibly be correct, could it? 
> Or is this 
> > > > normal. If needed I can post some stats and command 
> line options.
> > > 
> > > Why couldn't it?  Fast disks and CPU will easily overrun a 10Mb 
> > > connection for network file service.  Don't forget 3Mb/s is only 
> > > 300KB/s compared to 5-30MB/s for disks.
> > 
> > Sorry, I meant to emphasize the point that saturation occurred both 
> > ways, inbound and outbound. I can understand the saturation on the 
> > upload as it is transfering changed files up to the 
> destination, but 
> > what is it downloading that is so large?
> 
> I assume by upload you mean write and download, read.
> 
> stat(), readdir, and if --no-whole-file reading so it can do 
> checksums followed by reading in order to merge matched 
> blocks with new.
> 
> > > Your description leaves a couple of important things out:
> > > version of rsync and how rsync views the transfer with regard 
> > > to networking.
> > 
> > Rsync version 2.5.6. Both servers are on the same local segment.
> 
> As rsync views the transfer, not the physical layout.
> A local smbfs mount does not look to rsync like a network tranfer.
> 
> > 
> > > Make sure that rsync is doing a --whole-file transfer.
> > > Otherwise it will really thrash the destination.
> > 
> > I am not doing --whole-file transfer, but will try that for the 
> > session tonight.

Just to let you know, including the --whole-file made a huge difference.
The upload (write) speed was still at the 3mb level, but the read
(download) was now at less than 100k, instead of matching the upload
(write) speed.

Thanks for the help.

Max




More information about the rsync mailing list