rsync shows poor throughput vs. scp

[Tony Abernethy]

You should get some better answers, but a couple of points jump out at me.
If the files are already compressed, "small" changes result in very
different files,
so the business of reading both the target and the source to find common
stuff is kinda counterproductive.

Also the -z (compressing something already compressed) can't really do much
good,
particularly with LAN speeds

Methinks in this case, you want to disable a bunch of stuff that makes rsync
great. Seriously.
Basically, you want rsync to copy the entire file if the timestamp is
different.
You do not want thingees that are extremely useful for transferring big
files over bad connections.
(I suspect that here is a case where you use rsync because it's easier to
figure out rsync
that to use something (actually quite different) that is more "appropriate".

You discover that "copy stuff from here to there (or vice-versa) is actually
incredibly complicated as to exactly what you (should) mean.
  -----Original Message-----
  From: rsync-bounces+tony=servacorp.com at lists.samba.org
[mailto:rsync-bounces+tony=servacorp.com at lists.samba.org]On Behalf Of Marty
Mulligan
  Sent: Wednesday, May 24, 2006 2:29 PM
  To: rsync at lists.samba.org
  Subject: rsync shows poor throughput vs. scp


  hi all- been reading through the archives but I still can't seem to find a
solution to my problem.  I am using rsync to keep mirror copies of content
which is being served (via http) on both the sender and receiver.

  The files on average are 20-50mb each, (mostly already in a compressed
format... mp3, etc) and both sender and receiver pushes around 30mbps on
average.

  the sender is a Celeron 1.3GHz with 1GB RAM
  receiver is an AMD 1.2GHz with 1GB RAM

  Both have dual IDE drives configured for software RAID0

  When I initiate an rsync from the receiver, the file list is built in
around / under 1 minute, and the transfer proceeds at around 500K/s

  This seems very slow since the machines are on the same subnet and sitting
right next to each other in the same datacenter.  While this rsync is
running, I open a second terminal and do an scp, again initiated from the
receiver copying from the sender, and the transfer proceeds at around 5MB/s!

  Why is scp so much faster than rsync here?  Is there anything I can do to
improve the speed of these transfers?

  Fwiw, this is the rsync command I'm issuing:

  rsync -azL --whole-file --stats --progress --delete sender::my_files
/test_destination

  and rsyncd.conf on the sender looks like:

  use chroot = no
  max connections = 10
  pid file = /var/run/rsyncd.pid
  motd file = /etc/rsync/rsyncd.motd
  timeout = 300
  transfer logging = yes
  log file = /var/log/rsyncd.log
  log format = %t %h (%a) %m %l %b %o %f

  [my_files]
  uid = root
  path = /my_files
  max connections = 5
  read only = true
  hosts allow = 192.168.0.0/24
  dont compress = *
  list = false


  MUCH respect and appreciation to anyone who can help!

  Thanks,

  Marty
-------------- next part --------------
HTML attachment scrubbed and removed