state of the rsync nation? (revisited 6/2003 from 11/2000)

[Donovan Baarda]

On Sun, 2003-06-08 at 00:31, Jeff Kowalczyk wrote:
> I'm interested in these very questions (librsync-rsync relationship,
> remaining limitations of rsync, active prospects for ground-up rewrites),
> Google searches for rsync info have proved a little too vague due to the
> programs ubiquity. Much has certainly changed since this was written,
> could some people with knowledge in these areas could update martin's
> response for the state of rsync, June 2003? Thanks.

regarding librsync... It is still in sort-of-active development on
SourceForge by a variety of developers... a new release is waiting in
CVS for me to finally get around to releasing it, but I'm busy on a big
contract at the moment so its currently on hold pending some more
cygwin/win32 testing. It is in active use by projects like rdiff-backup.

AFAIK, rproxy is pretty much dead, and the only version that exists
depends on a very old version of libhsync. The closest thing to this
available now is the http proxy "proof of concept" with xdelta, but it's
radically different in many ways to the old rproxy (due to xdelta not
using signatures).

> On 13 Nov 2000, Jason Ozolins wrote:
> http://lists.samba.org/pipermail/rsync/2000-November/003147.html
> > Just a quick question: is the librsync contained within the rproxy
> > source code meant to be tracking the development of the mainstream
> > rsync, or is it a stripped-down thing meant only to support rproxy?
> 
> On 13 Nov 2000, Martin Pool Responded: Here's a quick history:
[...]
> rsync gave rise to Josh Macdonald's XDelta, which is optimized for the
> case where old and new versions are on the same machine, and so it can
> generate more efficient deltas.

xdelta is still under active development by Josh, and is evolving into a
fancy versioning virtual file system... an ideal back-end for something
like subversion. Josh tends to develop stuff with little fanfare, but
his code tends to be _very_ clean.

> tridge extracted the algorithm into librsync, which I renamed to libhsync
> when I changed the wire format.  The code currently checked in as librsync
> is in my opinion not very good.  It tries to make the algorithm available
> at various levels to programs that would like to use it, though the only
> user at the moment is rproxy.  rsync doesn't use libhsync -- possibly it
> never will, as we care enough about rsync performance that tighter
> integration is justified.  Well, if we were starting from scratch it might
> be separated out, but it's not worth doing it retrospectively now.
[...]

This is largely still true, except libhsync changed back to librsync and
now has its own project on SourceForge separate from the mostly defunct
rproxy. librsync itself has no "wire format", being just a general
purpose signature/delta/patch library implementing the rsync algorithm.

The comments about rsync never using libhsync/librsync are still true
for the foreseeable future. There are many things rsync includes that
are still missing from librsync, and the rsync implementation is very
tightly coupled, with many backwards compatibility issues. Even when
librsync reaches the point of being as good or better than rsync at
signature/delta/patch calculation, it would be a major task to "fit it
into" rsync.

rsync also has more active development, mostly in the form of
incremental feature additions and the resulting "bugfix fire-fighting",
all of which lead to an even more tangled implementation. Occasionally
there are efforts to re-write and clean up sections of the code, but
they are (rightly) regarded cautiously because of the breakage risk
involved for little immediate gain.

The librsync code in CVS is still largely "not very good". It is pretty 
messy and needs a good cleanup. The API is mostly OK though, and it
_does_ work quite well, with no known bugs. I have some plans for a
major cleanup and optimisation of the code based on my experiences with
pysync. I have a patch submitted that I plan to commit after the next
release that optimises and cleans up the delta calculation code quite a
bit.

The "next big thing" in delta calculation is probably going to be the
vcdiff encoding format, which should allow a common delta format for
various applications and supports "self-referencing delta's", which
makes it capable of compression. According to the xdelta project this
has already been implemented, and I'm keen to see Josh's code, as it
could be used as the basis for a cleanup/replacement of at least the
"patch" component of librsync.

Possibly worth also mentioning is things like pysync which is a
demonstration implementation of rsync and xdelta, as well as a wrapper
for librsync. I'm kind of embarrassed though that at the moment
rdiff-backup probably has a better python wrapper of librsync than
pysync does.

I believe there has also been some implementations of rsync in Perl (one
that claims to talk to rsync, which is an amazing achievement), but I'm
not up to date on those. I think someone has a Perl wrapper for librsync
that was being used as a test bed for rsync 3 type development
(superlifter?).

For the future I can see continued support of the exising rsync code. I
would also like to see librsync adopt vcdiff as it's delta format, and
get a major cleanup, possibly by re-using some xdelta code. There are
many common elements to the xdelta and rsync algorithms, and I see no
reason why a single library couldn't support both (as pysync does). It
would be nice if librsync and/or xdelta could become _the_ delta
library.

-- 
----------------------------------------------------------------
Donovan Baarda                http://minkirri.apana.org.au/~abo/
----------------------------------------------------------------