How to speed up rsync when haveing lots of files

[Daniel.Li]

On Tue, 2009-03-03 at 23:46 -0700, lewis butler wrote:
> On 3-Mar-2009, at 22:55, Daniel.Li wrote:
> > -z is apparently affect the performance when CPU has a lower  
> > frequency,
> > like 200MHz or so. When doing rsync, 100% cpu occupied, which limits
> > network performance.
> 
> You should not use -z at ALL with large numbers of small files.  The  
> increased latency of the compression/decompression will far exceed any  
> time saved in transmission.  You're not on a 300 baud modem, I assume?  
> -z should be used when you are sending large files that are  
> compressible in the first place (so not video or mp3s, for example),  
> but it it only going to hurt you on small files.
> 
> How small?  Depends on your connection.  My rule of thumb, based on  
> nothing at all, is that the speed in megabits of my connection is the  
> size in megabytes that I start worrying about compression.   
> Compressible data over 15MB in size?  OK, I will start to think about  
> using -z. If I was on a T1?  1.5MB.  ADSL? 700K. Arbitrary, perhaps,  
> but it seems to have served me well. In fact, I found removing -z sped  
> up transfers quite a bit on my LAN.

Humm,,, u mentioned about average file size and network connection,
which should taken into consideration of using "-z" option.

Well, I think when developing this option, it must have some report or
concerns. And after implement this feature, there will be also some kind
of test report, or some stuff like that to say "hello, this works". 

But I don't know where I can get those information:(

> 
> > BTW: I don't know more about "--compress-level=NUM", how many levels  
> > we
> > could use to set?
> 
> 
> See above. As for the number range, I've always assume a 0-9 range  
> much like gzip, though that is not explicitly given in the man page.
> 

Yeah, if we had above information, then it's much closer to the answer.
Even if we don't have the data, then we have the direction to test it
out.

BTW, currently, I have 100Mb local network. I need to get the best
result of rsync. Currently, my data shows that
total_file_size/time(include transfer and preparition), vary from
2.72~7.7MB/s

So I wanna know how to maximize the performance both of large file (1GB
patter file) and small file (MP3, about 2~5MB per file)

Maybe MP3 shouldn't be considered small files, right? 

Do u guys have any suggestions? 
Any hint or advice is mostly appreciated. Thanks.

-- 
Daniel Li