How to speed up rsync when haveing lots of files

[lewis butler]

On 3-Mar-2009, at 22:55, Daniel.Li wrote:
> -z is apparently affect the performance when CPU has a lower  
> frequency,
> like 200MHz or so. When doing rsync, 100% cpu occupied, which limits
> network performance.

You should not use -z at ALL with large numbers of small files.  The  
increased latency of the compression/decompression will far exceed any  
time saved in transmission.  You're not on a 300 baud modem, I assume?  
-z should be used when you are sending large files that are  
compressible in the first place (so not video or mp3s, for example),  
but it it only going to hurt you on small files.

How small?  Depends on your connection.  My rule of thumb, based on  
nothing at all, is that the speed in megabits of my connection is the  
size in megabytes that I start worrying about compression.   
Compressible data over 15MB in size?  OK, I will start to think about  
using -z. If I was on a T1?  1.5MB.  ADSL? 700K. Arbitrary, perhaps,  
but it seems to have served me well. In fact, I found removing -z sped  
up transfers quite a bit on my LAN.

> BTW: I don't know more about "--compress-level=NUM", how many levels  
> we
> could use to set?


See above. As for the number range, I've always assume a 0-9 range  
much like gzip, though that is not explicitly given in the man page.

-- 
The quality of our thoughts and ideas can only be as good as the quality
     of our language.