Performance problem

[Andrew Boyce-Lewis]

On Mon, 2004-01-12 at 22:15, jw schultz wrote:
> On Mon, Jan 12, 2004 at 06:22:04PM -0500, Andrew Boyce-Lewis wrote:
> > Hi, I am running rsync version 2.5.7 (stock distro on redhat linux, ES
> > and 9) to rsync a directory with ~300k files in it from a machine on a
> > 10Mbit internet tap to a machine with a 100Mbit internet tap. The
> > problem is that I am only getting about 500Kbps during the transfer. I
> > have tested link speed by scping a file between the two systems,
> > performance was approximately 7.5Mbit/sec.  Ping time between the two
> > locations is about 2.5 ms (only 20 miles apart)
> > 
> > This is obviously a problem. I have also tried rsyncing just a few files
> 
> It is?  Rsync and scp do two completely different things and
> their performance is not comparable.

Not comparable, but seriously... 62 KB/sec vs 960KB/sec? I have done
similar operations over gig-e with much better performance, on almost
exactly the same hardware. 

The transfers that I am having rsync do are of whole, large files (say 5
- 15 MB each), so I would think that scp and rsync (in this case) would
actually be somewhat comparable. 

> 
> > (say 400) to rule out problems related to huge file lists, however I had
> > the same problem. 
> > 
> > Is this a known problem? Am I missing something really obvious?
> > 
> > I have tried testing drive performance in each machine and have found it
> > to be acceptable. 
> 
> But how is the filesystem performance at doing what rsync
> does to it?  Raw sequential throughput is irrelevant.

Well, with out good hard data to back this up, the drives at both ends
are ultra 160 scsi raids, and hdparm suggests about 40MB/sec write (yes,
I know hdparm is a bad benchmark). I have seen no signs of any disk
performance problems in other applications.

> 
> > The line that I am using to do the rsync is the following:
> > 
> > rsync -avvu --exclude-from=/root/sync_list.exclude --delete -e ssh
> > --stats /data/ dest:/data/
> 
> Try --whole-file;  if that improves things you are disk bound.
> 
> Drop to just one -v; -vv is for debugging and adversely
> affects network buffering.
I will give both a try this evening.

> 
> It probably isn't your exclude file but a large exclude file
> would add overhead.

the exclude file has only two entries. 

> 
> 2.5.7 is a security release with minimal changes from 2.5.6
> which is almost a year old.  There have been a lot of
> changes since then most of which are in 2.6.0.  Since 2.6.0
> there have been a number of changes related to performance
> including one that dramatically improves the network
> performance.  All of this you would have known if you had
> read the list archives or checked NEWS.

I did read the archives and I did read the NEWS. I am aware that 2.6.0
is newer, but I am also aware that the performance problems that I am
having are abnormal. The fact of the matter is that 2.6.0 has not been
widely adpoted yet. I am not opposed to running 2.6.0, but I would at
least like to try to figure out whats wrong with 2.5.7, especially since
the problem that I am having is NOT in the known bugs and is NOT
addressed in the archives. 

 -Andrew
> 
> 
> -- 
> ________________________________________________________________
> 	J.W. Schultz            Pegasystems Technologies
> 	email address:		jw at pegasys.ws
> 
> 		Remember Cernan and Schmitt
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://lists.samba.org/archive/rsync/attachments/20040113/bc591c73/attachment.bin