[PATCH] Batch-mode rewrite

[Chris Shoemaker]

On Thu, Jul 22, 2004 at 06:36:27PM -0700, Wayne Davison wrote:
> On Wed, Jul 21, 2004 at 03:54:11PM -0400, Chris Shoemaker wrote:
> > What data exactly?  I thought:
> > 	1) all recv-to-gen communications went through the error_pipe[] fds.
> 
> Yes, that became true when I got rid of the extra pipe that used to
> separate the redo values from the error messages.
> 
> > 	2) the only meaningful communications were redo requests and
> > 	"I'm done".
> 
> It depends on which side is the client.  Since we made the read-batch
> side always the client then the errors/warnings get output to the tty
> directly (for a client sender, which is the default for non-batch local
> transfers, the pipe also contains any error messages from the receiver).
> There is also a patch in the patches dir that sends a "success" message
> for every correctly transferred file (delete-sent-files.diff).
> 
> > I thought we could skip the redos and fake the "I'm done".
> 
> Yes, it's possible to turn off the communication, which is what I was
> talking about -- I was mentioning what things would need to be
> special-cased.
> 
> > Ah, I see what you mean about the directory tweaks.
> 
> And don't forget the hard-link post-processing -- it would also need to
> happen after the receiver finished its job.

Ok, it's late and this new patch seems like the simplest solution of
all, so there's probably something very wrong with it.  Basic idea:
move hard-link post-processing and directory permission tweaks from
generate_files() to recv_files().  Then there's no syncing issues,
because receiver cleans up when receiver's done.  Uh, right?  :)

Even with the index notification, this kind of decoupling of recv and
generator makes sense, don't you think?

> 
> > Just to clarify, I don't have anything against the index notification
> > style gen/recv syncronization.  If you think that's better, then let's
> > go that way.
> 
> In the future there may be a need for data to be communicated back from
> the generator to the receiver (i.e. if the --fuzzy patch get improved
> enough to make it worthwhile), so I think in the long run that having
> the two things running in parallel (like normal) will be the easiest
> thing to maintain.  We can certainly revisit the issue later, but for
> now I have the code in the generator sending the normal index bytes down
> the output pipe (it just avoids sending the checksum data), and some
> special-case read-batch code in the receiver that reads the from-the-
> generator pipe and only proceeds with an update if the generator said
> it was time.
> 
> One interesting thing that fell out of this was the ability to skip any
> part of a batched update that had already been done.  For instance, if
> you run the ./BATCH.sh file and terminate it (for some reason), you can
> run it again and it will skip all the already-performed updates.  (One

That's a pretty compelling feature.  BTW, your index notification scheme
isn't in CVS, is it?

-chris

> exception: if --partial had been specified the interrupted file will
> probably not update correctly, but rsync should tell you when that
> happens.)
> 
> ..wayne..
-------------- next part --------------
Index: generator.c
===================================================================
RCS file: /cvsroot/rsync/generator.c,v
retrieving revision 1.102
diff -u -r1.102 generator.c
--- generator.c	21 Jul 2004 23:59:25 -0000	1.102
+++ generator.c	23 Jul 2004 03:37:49 -0000
@@ -593,8 +593,13 @@
 
 	write_int(f_out, -1);
 
-	if (preserve_hard_links)
-		do_hard_links();
+	if (verbose > 2)
+		rprintf(FINFO,"generate_files finished\n");
+}
+
+void restore_directory_perms(struct file_list *flist, char *local_name)
+{
+	int i;
 
 	/* now we need to fix any directory permissions that were
 	 * modified during the transfer */
@@ -606,6 +611,4 @@
 			       file, i, -1);
 	}
 
-	if (verbose > 2)
-		rprintf(FINFO,"generate_files finished\n");
 }
Index: receiver.c
===================================================================
RCS file: /cvsroot/rsync/receiver.c,v
retrieving revision 1.96
diff -u -r1.96 receiver.c
--- receiver.c	22 Jul 2004 15:31:06 -0000	1.96
+++ receiver.c	23 Jul 2004 03:37:50 -0000
@@ -575,12 +575,17 @@
 				send_msg(MSG_REDO, buf, 4);
 			}
 		}
-	}
+	} /* end of while(1) */
 	make_backups = save_make_backups;
 
 	if (delete_after && recurse && !local_name && flist->count > 0)
 		delete_files(flist);
 
+	if (preserve_hard_links)
+		do_hard_links();
+
+	restore_directory_perms(flist, local_name);
+
 	if (verbose > 2)
 		rprintf(FINFO,"recv_files finished\n");