How to speed up rsync when haveing lots of files

Boniforti Flavio flavio at piramide.ch
Wed Mar 4 07:26:18 GMT 2009


> You should not use -z at ALL with large numbers of small 
> files.  The increased latency of the 
> compression/decompression will far exceed any time saved in 
> transmission.  You're not on a 300 baud modem, I assume?  
> -z should be used when you are sending large files that are 
> compressible in the first place (so not video or mp3s, for 
> example), but it it only going to hurt you on small files.

I was lurking your thread, when I stumbled across your considerations
about "-z" option. I'm having issues with slow network, which I presume
is the main cause of failure in rsnapshot sync tasks.
Do you suggest to try *without* compression to see if something works
better?

> How small?  Depends on your connection.  My rule of thumb, 
> based on nothing at all, is that the speed in megabits of my 
> connection is the  
> size in megabytes that I start worrying about compression.   
> Compressible data over 15MB in size?  OK, I will start to 
> think about using -z. If I was on a T1?  1.5MB.  ADSL? 700K. 
> Arbitrary, perhaps, but it seems to have served me well. In 
> fact, I found removing -z sped up transfers quite a bit on my LAN.

Uhm... It's quite interesting the way you somehow "arbitrarily" choose
when to worry about compression....

Is there any way to know *in advance* if using or not using "-z" could
be the better solution?

TIA,
Flavio.


More information about the rsync mailing list