superlifter design notes and a new proposal

[Donovan Baarda]

On Tue, Aug 06, 2002 at 02:45:47PM +1000, Martin Pool wrote:
> I was talking to Tim the other day about the backup vs transfer
> dichotomy identified by John.
> 
> One interesting idea, further down the track, would be to introduce a
> virtual filesystem layer into rsync 3, similar to that in Samba, so
> that all disk IO goes through a function-table layer.
> 
> You could then write a filesystem layer that, rather than talking to
> the native filesystem, talks to some kind of database.  This would be
> very nice for a backup server for a couple of reasons.  (It could be a
> Sleepycat DB or SQL or whatever.)
> 
> Firstly, the database could implement arbitrarily rich filesystem
> semantics without being limited by the native filesystem.  For
> example, you couldn't back up OS X resource forks to a Solaris
> machine, but you can easily store them inside a database.  Similarly
> for storing extended attributes, or storing NT security descriptors on
> Unix (or vice versa).
> 
> Secondly, the database could be tuned for the particular case of
> storing incremental backups: between one version of a file and the
> next, you could just store an xdelta diff, and identical files could
> be replaced by something similar to hardlinks.  The whole thing could
> be compressed.  Basically you would be tuning for the case of an
> append-only filesystem.

Hmmm, sounds like you are re-inventing xdelta2... uses a bdb backend, has
support for extended attributes, and even has an NFS interface...

If you are considering this, I thoroughly recommend looking at xdelta2
first. Probably worth speaking to jmacd too, because he has already been
there and will probably have comments about ways he could make it better.

-- 
----------------------------------------------------------------------
ABO: finger abo at minkirri.apana.org.au for more info, including pgp key
----------------------------------------------------------------------