checksum-xattr.diff [CVS update: rsync/patches]

[Matt McCutchen]

On 7/2/07, Wayne Davison <wayned at samba.org> wrote:
> On Mon, Jul 02, 2007 at 08:43:39AM -0400, Matt McCutchen wrote:
> > What do you mean?  There's no way to fix the example I gave with
> > xattrs
>
> Not so.  I went on to explain how that is possible in my prior email
> (i.e. avoiding caching a checksum on a "now" mtime file is all that is
> needed).

I do not see how avoiding caching a checksum on a "now" mtime file
affects my example.  In my example, a file is written at one time (say
5) and gets mtime 5.  At some *later* time (say 8), rsync runs and
caches the checksum, which sets the ctime to 8.  The trouble is that,
right after rsync caches the checksum, someone could quickly modify
the file and then set the mtime back to 5.  If he manages to do this
before time ticks to 9, the ctime will still be 8, and rsync will have
no way of knowing the file was modified.

It still appears to me that this is one case that xattr-based checksum
caching cannot be made to handle correctly.  That doesn't mean anyone
should care; the case is unlikely and setting a file's mtime back
after modifying it is a silly thing to do.

> > That's easy to fix: get your "now" by touching a file on the
> > filesystem and reading the resulting mtime.
>
> Yes, that's one solution that I had already thought of.  You'd also need
> to do that for every filesystem in the transfer, so you need to add
> filesystem checking and hope that you always have write permissions to
> the dirs holding the files (or have a work-around algorithm if you
> don't).  As I said, it's complicated (and quite a bit of hassle).

It is a hassle.  To make matters worse, the mapping between st_dev
numbers and filesystems might change over time.  The same filesystem
could get a different st_dev if the hardware is connected to a
different port.  More troublingly, I could prepare two external disks
that are identical except for the actual contents of the files, mount
one, let rsync cache its checksums, and then switch to the other; the
second disk would get the same st_dev and rsync would be fooled.

The approach I'm considering is to have a separate cache file for each
filesystem and store it on the filesystem itself.  That avoids the
issues with st_dev and has the added benefit that, if a disk is moved
to a different computer, the checksums come with it for that computer
to use.  Conveniently, rsync could touch the cache file itself to get
"now" and possibly run other tests on the cache file to make sure the
behavior of the filesystem's mtimes and ctimes is sane enough to make
correct caching possible.

I propose that sysadmins give each user a directory on each filesystem
at a standard path from the filesystem root to hold the cache
files.(*)  When rsync stats a file and sees a st_dev that doesn't
match any of the caches it already has open, it could obtain the mount
point of that st_dev (I'm not sure how but I'm sure it's possible) and
follow the appropriate path relative to the mount point to open the
cache or create a new one.

* Storage of checksum caches is just one of several uses for a
per-user, per-filesystem "staging directory".  Such a directory could
also be used as a secure rsync --temp-dir and could contain a trash
area emptied by a setuid-root program (so users can delete others'
nonempty directories from their dropboxes).  At one point, my computer
provided staging directories at "{/,/home/}staging/$UID".

> Changing my assumed function-calling order to be first time() then
> stat() would take care of this, since any roll-over in the stat() mtime
> would appear to be in the future, and would be seen as being too recent.

Yes, I had been assuming time() (actually, stat() on the touched file)
before stat() on the data file.

To economize on system calls, I propose a single time() at the
beginning.  If a stat() then reveals a "now" mtime, I propose calling
time() again to update "now".  If it changed and another stat()
confirms that the file is still older than the new "now", the file's
checksum can be cached.

> I don't see any need for that for the xattr version, since rsync isn't
> going to update the checksums (just optionally create them on its temp
> files).

OK.

> For the non-xattr version it would be nice to have a better
> cache mechanism than the simple per-dir .rsyncsums files I implemented
> in my patch: having a library that implemented a checksum lookup/update
> by dev+inode using a global checksum cache would be cool, and avoid the
> file droppings.  Making it so that different programs could request a
> checksum of a particular type concurrently (which the server/library
> would return from cache, if possible, or compute and store in the cache,
> if safe) would make it generally useful for a variety of programs.  That
> would be quite easy for rsync to support, if it existed.

OK, I think I will start working on such a library.  I don't know
whether it will be ready in time for the release of rsync 3.0.0; until
it is ready, rsync could keep the existing .rsyncsums support or drop
it and provide only the xattr support.

Matt