superlifter design notes and a new proposal

[Martin Pool]

I was talking to Tim the other day about the backup vs transfer
dichotomy identified by John.

One interesting idea, further down the track, would be to introduce a
virtual filesystem layer into rsync 3, similar to that in Samba, so
that all disk IO goes through a function-table layer.

You could then write a filesystem layer that, rather than talking to
the native filesystem, talks to some kind of database.  This would be
very nice for a backup server for a couple of reasons.  (It could be a
Sleepycat DB or SQL or whatever.)

Firstly, the database could implement arbitrarily rich filesystem
semantics without being limited by the native filesystem.  For
example, you couldn't back up OS X resource forks to a Solaris
machine, but you can easily store them inside a database.  Similarly
for storing extended attributes, or storing NT security descriptors on
Unix (or vice versa).

Secondly, the database could be tuned for the particular case of
storing incremental backups: between one version of a file and the
next, you could just store an xdelta diff, and identical files could
be replaced by something similar to hardlinks.  The whole thing could
be compressed.  Basically you would be tuning for the case of an
append-only filesystem.

At a more modest level, a VFS would be a clean way to pluggably
implement storing EAs in a .ea directory.

This is pie in the sky of course, but perhaps eventually a new cleaner
architecture would make it possible.

-- 
Martin