atomic transaction set option to rsync
Dag Wieers
dag at wieers.com
Tue Jan 4 01:51:23 GMT 2005
On Mon, 3 Jan 2005, Wayne Davison wrote:
> On Mon, Jan 03, 2005 at 05:39:19PM +0100, Dag Wieers wrote:
> > With newer rsyncs, rsync seems to sort the transaction set, so I
> > cannot longer use this trick to have the metadata uploaded just after
> > the data in the same transaction set.
>
> Rsync has always sorted the list of files to be sent, so this is not
> something that is different between 2.5.x and 2.6.x. I'd be interested
> in hearing what you believe to be different in how files are processed.
In the past I could say smt. like:
rsync -a dir1/ dir2/ user at rsync:/remote-dir/
and it would process first dir1 and then dir2. I'm not sure when this
change happened, it may not be when 2.6.x.
This way I first added the packages/ dir (which contains hardlinks of only
the packages) and then the repository (packages+metadata). Resulting in a
much smaller window between the start of mirror the repodata and finishing
the hardlinking of all packages. (still 5 to 10 minutes !)
If rsync includes the atomic transaction set I explained in a previous
post, it can be reduced to an even smaller window (only the renaming of
the files strictly). Even the hardlinking can be done in advance.
> > I was wondering if it was possible and acceptable to have an rsync
> > option to update the whole transaction in a atomic (or near-atomic
> > way).
>
> One way to do this would be to use the --link-dest option to create a
> new hierarchy of files (with only the changed files getting sent, and
> all unchanged files being hard-linked to the prior files) and then
> moving the whole file-set into place all at once. Imagine that there
> is a hierarchy you want to update in /dest/cur by running this script:
I know this, but I don't control all the mirrors and I can't ask all my
mirrors to implement one of these complex alternatives. Besides this
requires a per mirror configuration (as it requires a fixed directory per
rsync transaction). Something inherent to rsync (the same way current
atomic-per-file behaviour works) that kicks in at the end of a transaction
set instead of after each transfer (of an individual file) works
everywhere with a single option and no fuss.
I can ask mirrors to use '--atomic' or '--atomic-ts', bt I can't ask them
to re-organise their mirror-scripts just for me.
-- dag wieers, dag at wieers.com, http://dag.wieers.com/ --
[all I want is a warm bed and a kind word and unlimited power]
More information about the rsync
mailing list