Resolving problems in the generator->receiver pipes

jw schultz jw at
Thu Jan 15 20:23:16 GMT 2004

On Thu, Jan 15, 2004 at 12:11:39PM -0800, Wayne Davison wrote:
> When I was working on the the hard-link change, I noticed that many
> of the hard-link verbose messages were getting lost.  These messages
> get output very near the end of the transfer, and it turns out that
> the reason for the loss was that there are two pipes flowing from
> the generator and the receiver, and it was possible for the "we're
> all done" message to get received down the redo pipe without all of
> the messages getting sent down the error pipe.  It's also a long-
> known bug that the redo pipe can clog (but only if a really large
> number of redo items build up).
> In looking at this code, I figured out that it would be much better
> to just dump the redo pipe and use the error pipe as a message pipe.
> Since it already uses the same transfer protocol as the multiplexed
> IO between other components, it turned out to be a very simple thing
> to add a REDO and a DONE message to the existing code.  This gets
> rid of the redo pipe and all of its problems.
> At the same time I though it would be good to separate the existing
> message-sending code out of the log-file code.  This is because one
> other non-logging message is already supported (the "data" stream in
> multiplexed IO) so the addition of two extra messages makes it a
> better fit for the io.c code than the log.c code (i.e. when the log
> code needs to send a message to another program, it calls the code
> in io.c to do it instead of doing it itself).
> Finally, the flushing code was tweaked to allow it to signal that
> the writing out of the receiver->generator pipe should be completely
> flushed (from the in-memory list of messages).  This allows the
> final flushing to complete more efficiently than calling io_flush()
> in a loop.
> In my exuberance for having finally solved the redo-hang problem in
> a much better way than my previous suggested fixes for it, I went
> ahead and checked in my changes.  However, feedback is still
> gratefully accepted.  The changes can be found here:

What i looked at from the cvs digest looked like decent
code.  It was just lacking the overview you provided here.

This sounds pretty good.  Just needs some thorough testing.

	J.W. Schultz            Pegasystems Technologies
	email address:		jw at

		Remember Cernan and Schmitt

More information about the rsync mailing list