[librsync-devel] Re: state of the rsync nation? (revisited 6/2003 from 11/2000)

[Martin Pool]

On  8 Jun 2003, Donovan Baarda <abo at minkirri.apana.org.au> wrote:

> regarding librsync... It is still in sort-of-active development on
> SourceForge by a variety of developers... a new release is waiting in
> CVS for me to finally get around to releasing it, but I'm busy on a big
> contract at the moment so its currently on hold pending some more
> cygwin/win32 testing. It is in active use by projects like rdiff-backup.

> AFAIK, rproxy is pretty much dead, and the only version that exists
> depends on a very old version of libhsync. The closest thing to this
> available now is the http proxy "proof of concept" with xdelta, but it's
> radically different in many ways to the old rproxy (due to xdelta not
> using signatures).

The main reason why rproxy is dead is that dynamically-generated HTML
files, where in principle rproxy does best, are just not a very
important problem for many people at the moment.

For users with ADSL or better HTML is not a problem, binaries may be.  

I realize there are people in the world who are still using modem
connections where rproxy might be a win but I don't see any of them
asking for commit access.  

A positive thing for rproxy is that both Mozilla and Apache2 are
pretty stable now, and they have good interfaces for adding streaming
compression support.  Neither Apache1.2 or Netscape, which were
dominant when I started on it could handle this very well.

> This is largely still true, except libhsync changed back to librsync and
> now has its own project on SourceForge separate from the mostly defunct
> rproxy. librsync itself has no "wire format", being just a general
> purpose signature/delta/patch library implementing the rsync algorithm.
> 
> The comments about rsync never using libhsync/librsync are still true
> for the foreseeable future. There are many things rsync includes that
> are still missing from librsync, and the rsync implementation is very
> tightly coupled, with many backwards compatibility issues. Even when
> librsync reaches the point of being as good or better than rsync at
> signature/delta/patch calculation, it would be a major task to "fit it
> into" rsync.

I think it's best at the moment to let rsync continue as a nice stable
program, good at what it does.  Wayne and myself have been toying with
replacements in the background and perhaps in the future something
better will come out, and perhaps it will use librsync.

> rsync also has more active development, mostly in the form of
> incremental feature additions and the resulting "bugfix fire-fighting",
> all of which lead to an even more tangled implementation. Occasionally
> there are efforts to re-write and clean up sections of the code, but
> they are (rightly) regarded cautiously because of the breakage risk
> involved for little immediate gain.

> The librsync code in CVS is still largely "not very good". It is pretty 
> messy and needs a good cleanup.

True. :-/

I think I got a bit mentally twisted up by trying to support
nonblocking operation, which I still think is very important the the
library being generally useful.  Doing this in C is a bit hard.  But
it could certainly be done much better.

The other thing I would really like to see is a more thorough test
suite.

> The API is mostly OK though, and it _does_ work quite well, with no
> known bugs. I have some plans for a major cleanup and optimisation
> of the code based on my experiences with pysync. I have a patch
> submitted that I plan to commit after the next release that
> optimises and cleans up the delta calculation code quite a bit.
> 
> The "next big thing" in delta calculation is probably going to be the
> vcdiff encoding format, which should allow a common delta format for
> various applications and supports "self-referencing delta's", which
> makes it capable of compression. According to the xdelta project this
> has already been implemented, and I'm keen to see Josh's code, as it
> could be used as the basis for a cleanup/replacement of at least the
> "patch" component of librsync.

Yes, that sounds good.

> I think someone has a Perl wrapper for librsync that was being used
> as a test bed for rsync 3 type development (superlifter?).

"superlifter" was my prototype.  It uses Python, and in fact just
calls out to rdiff at the moment.

At the moment I see it as another layer above librsync/rdiff that
provides pipelined delta-compressed remote network IO, optionally over
SSL or SSH.  On top of this you could build a batch transfer like
rsync 2.6, or an interactive client, or a backup system like
Duplicity, or a real-time mirror based on dnotify.

> For the future I can see continued support of the exising rsync code. I
> would also like to see librsync adopt vcdiff as it's delta format, and
> get a major cleanup, possibly by re-using some xdelta code. There are
> many common elements to the xdelta and rsync algorithms, and I see no
> reason why a single library couldn't support both (as pysync does). It
> would be nice if librsync and/or xdelta could become _the_ delta
> library.

I heartily agree.

--
Martin