Optimizing RSync algorithm using techniques with Google used in courgette
Shachar Shemesh
shachar at shemesh.biz
Tue Sep 8 06:27:03 MDT 2009
Hasanat Kazmi wrote:
> Hi,
> I am student at LUMS SSE (http://cs.lums.edu.pk) and an active RSync user.
> Just few days ago, Google wrote about Courgette*: an algorithm which
> is specially written for syncing executables. By using Courgette,
> Google made diff size 1/10th of previous techniques used.
> I was wondering if this (or something on same lines) can be used to
> optimize RSync? I am senior and have to do a project. I am thinking to
> implement this in RSync. I need input from developers. What do you
> guys think?
>
> *http://dev.chromium.org/developers/design-documents/software-updates-courgette
>
> Hasanat Kazmi
>
Hi Hasanat,
Like you said in the subject, this is an optimization. A format specific
optimization. In other words, it uses a known property of the file being
synchronized in order to make the diff size smaller. If you were to try
to use the courgette pre-processing on something which is not an
executable, you would have gotten significantly worse results than
merely running rsync.
At the moment, for better or for worse, rsync does not do format
specific optimizations. As long as that is the case, rsync cannot be
optimized using this algorithm.
Even if we (and by "we", I mean Wayne, or anyone else brabe enough to
pick this task up) were to implement such a functionality, I can think
of quite a few things that would have a lot more to gain than
executables. In particular, something that would uncompress both source
and destination, and apply the rsync algorithm to both files, and then
make sure the recompression of the target produces the exact same result
would, IMHO, be much more useful than the change you are suggesting.
Shachar
--
Shachar Shemesh
Lingnu Open Source Consulting Ltd.
http://www.lingnu.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20090908/296a5007/attachment.html>
More information about the rsync
mailing list