Rsync friendly zlib/gzip compression - revisited

[Kevin Day]

Per my comment yesterday:

> However, I just kicked off another parametric study varying the rsync block size and the block mask size, while holding the window size constant.  If there is an optimization avaialble for rsync block size, then we should see a dependency in these 
results.


The parametric test shows that smaller rsync blocks result in better performance (as expected), and that there is a nice, straightline linear dependency on it.  Mask size shifts the line up or down, but does not change the shape.

My conclusion:  There probably is not an optimal rsync block size for this type of file - smaller will always be better (unless we take the size of the rsync checksum table into account, in which case we are back to the results of Andrew's original 
thesis).

The size of the rsync block provides a first order effect on the algorithm's efficiency.  The size of the compression block mask size has a second order effect, but the effect is definitely there.

I'll be implementing the rsync checksum in the zlib algorithm next, to see if going to really small window sizes improves the performance the way I think it will.

Cheers,

- K