Preliminary Suggestion For Atomic Transactions

[Wayne Davison]

On Thu, Jan 06, 2005 at 02:59:44AM +0800, Jeff Pitman wrote:
> The attached patch (extremely ugly) attempts to resolve this by
> foregoing the rename step until the end.

Thanks for the patch.  It looks like it implements the basics of the
idea, but I think that it will be better to use deterministic temp-file
names like the --partial-dir logic gives us.  I also chose to use an
option name of --delay-renames.  So, I've created my own version of this
post-transfer renaming heuristic, which is available from here:

http://rsync.samba.org/ftp/unpacked/rsync/patches/delay-renames.diff

The patch is relative to the very latest CVS or "nightly" tar file.

One unresolved issue I thought of is how rsync should deal with an
absolute --partial-dir pathname:  the above implementation woul put
all the partial files into that one dir (a relative pathname, like
the default, ensures that the files get sprinkled around).

> The first big [problem] in running the testsuite is that the parent
> dirs mtime is not getting preserved.  [...] Next, hardlinks need some
> work.

These problems stem from a bug in the synchronization between the
receiver and the generator that I just discovered:  the generator does
not wait until all the redo & delete-after work is done before it begins
the final hard-link pass and the final directory-time-tweaking pass.
Because of this, the extra renaming loop added at the end of the
receiver's work didn't get done before the generator got going on the
hard-linking and directory-time tweaking.  I have checked-in a fix for
this, so the above delay-renames.diff works fine with a test-suite that
has been modified to use --delay-renames on all the possible commands.

Here's some of the design items that the patch deals with:

  - Complain if the user combines --inplace with --delay-renames.

  - A client sender needs to pass the new option to a server receiver.

  - We use the --partial-dir logic to make all the temp-file names
    deterministic, using the default name of ".~tmp~" if the user did
    not specify the --partial-dir option.  Note that these dirs get
    auto-excluded by the normal --partial-dir handling, so they will not
    be transferred by default, and they will not be deleted by --delete
    or --delete-after if the prior transfer did not complete and some
    temp files got left around (they will also be used to speed up the
    transfer when it is restarted).

The patch also fixes/changes a few things from your patch:

  - The rename loop must happen prior to the delete-after logic in the
    receiver.

  - It is good to avoid twiddling the flist->flags bits in the forked
    receiver because it will cause all the shared file-list memory
    (between the generator and the receiver) to become unshared (on
    systems that use copy-on-write memory when forking).

  - It is good to avoid adding large amounts of memory to the flist
    object (adding MAXPATHLEN chars of extra memory for each file
    transferred would bloat rsync's memory by a huge amount for large
    transfers).

I've attached a 3-hunk patch that kluges in the use of the new
--delay-renames option into all the (relevant) test-suite rsync
commands (for temporary testing).  Apply it after the main patch,
if desired.

..wayne..
-------------- next part --------------
--- options.c	2005-01-10 10:16:54 +0000
+++ options.c	2005-01-10 10:30:07 +0000
@@ -456,6 +456,7 @@
   {"protocol",         0,  POPT_ARG_INT,    &protocol_version, 0, 0, 0 },
   {"server",           0,  POPT_ARG_NONE,   &am_server, 0, 0, 0 },
   {"help",            'h', POPT_ARG_NONE,   0, 'h', 0, 0 },
+  {"delay-renames",    0,  POPT_ARG_NONE,   0, 0, 0, 0 },
   {0,0,0,0, 0, 0, 0}
 };
 
@@ -947,6 +948,8 @@
 			bwlimit_writemax = 512;
 	}
 
+	if (inplace)
+		delay_renames = 0;
 	if (delay_renames && !partial_dir)
 		partial_dir = ".~tmp~";
 
--- runtests.sh	2004-02-04 05:34:29 +0000
+++ runtests.sh	2005-01-10 10:27:33 +0000
@@ -168,7 +168,7 @@
     exit 2
 fi
 
-RSYNC="$rsync_bin"
+RSYNC="$rsync_bin --delay-renames"
 
 export rsync_bin RSYNC