atomic transaction set option to rsync

Dag Wieers dag at wieers.com
Tue Jan 4 01:51:23 GMT 2005


On Mon, 3 Jan 2005, Wayne Davison wrote:

> On Mon, Jan 03, 2005 at 05:39:19PM +0100, Dag Wieers wrote:
> > With newer rsyncs, rsync seems to sort the transaction set, so I
> > cannot longer use this trick to have the metadata uploaded just after
> > the data in the same transaction set.
> 
> Rsync has always sorted the list of files to be sent, so this is not
> something that is different between 2.5.x and 2.6.x.  I'd be interested
> in hearing what you believe to be different in how files are processed.

In the past I could say smt. like:

	rsync -a dir1/ dir2/ user at rsync:/remote-dir/

and it would process first dir1 and then dir2. I'm not sure when this 
change happened, it may not be when 2.6.x.

This way I first added the packages/ dir (which contains hardlinks of only 
the packages) and then the repository (packages+metadata). Resulting in a 
much smaller window between the start of mirror the repodata and finishing 
the hardlinking of all packages. (still 5 to 10 minutes !)

If rsync includes the atomic transaction set I explained in a previous 
post, it can be reduced to an even smaller window (only the renaming of 
the files strictly). Even the hardlinking can be done in advance.


> > I was wondering if it was possible and acceptable to have an rsync
> > option to update the whole transaction in a atomic (or near-atomic
> > way).
> 
> One way to do this would be to use the --link-dest option to create a
> new hierarchy of files (with only the changed files getting sent, and
> all unchanged files being hard-linked to the prior files) and then
> moving the whole file-set into place all at once.  Imagine that there
> is a hierarchy you want to update in /dest/cur by running this script:

I know this, but I don't control all the mirrors and I can't ask all my 
mirrors to implement one of these complex alternatives. Besides this 
requires a per mirror configuration (as it requires a fixed directory per 
rsync transaction). Something inherent to rsync (the same way current 
atomic-per-file behaviour works) that kicks in at the end of a transaction 
set instead of after each transfer (of an individual file) works 
everywhere with a single option and no fuss.

I can ask mirrors to use '--atomic' or '--atomic-ts', bt I can't ask them 
to re-organise their mirror-scripts just for me.

--   dag wieers,  dag at wieers.com,  http://dag.wieers.com/   --
[all I want is a warm bed and a kind word and unlimited power]


More information about the rsync mailing list