benchmarking rsync's -z compression utility

[Donovan Baarda]

On Sat, 2003-05-10 at 19:58, Leaw, Chern Jian wrote:
> Hi,
> Is there a way in which rsync's -z compression (zlib) utility can be
> benchmarked? 
> 
> I'm trying to compare the compression ratio between rsync and external
> compression tools like gzip and bzip2. 
> 
> Are there any advantages to using rsync's internal compression mechanism
> specified with the -z option compared to solely applying external
> compression i.e. bzip2 to the files and invoking rsync to transfer these
> files without the -z option?

I'm assuming here you are talking about using librsyncs -z vs running
librsync without it through a compressed pipe, and are aware that rsync
does delta-compression to updated a basis file in both cases.

rsync _should_ be able to do better with -z because it uses
"context-compression" by "priming" the compressor with hits and
discarding the compressed output. This means the compressor and
de-compressor see the whole file, even though only the compressed miss
data is transmitted.

my experiments with pysync confirmed that this does make a measurable
difference (see the comments with pysync) on real world compressible
data.

A similar benefit could be achieved with self-referencing deltas, as
supported by the vcdiff format (soon to be) used by xdelta.

-- 
----------------------------------------------------------------
Donovan Baarda                http://minkirri.apana.org.au/~abo/
----------------------------------------------------------------