Resolving problems in the generator->receiver pipes

[Wayne Davison]

When I was working on the the hard-link change, I noticed that many
of the hard-link verbose messages were getting lost.  These messages
get output very near the end of the transfer, and it turns out that
the reason for the loss was that there are two pipes flowing from
the generator and the receiver, and it was possible for the "we're
all done" message to get received down the redo pipe without all of
the messages getting sent down the error pipe.  It's also a long-
known bug that the redo pipe can clog (but only if a really large
number of redo items build up).

In looking at this code, I figured out that it would be much better
to just dump the redo pipe and use the error pipe as a message pipe.
Since it already uses the same transfer protocol as the multiplexed
IO between other components, it turned out to be a very simple thing
to add a REDO and a DONE message to the existing code.  This gets
rid of the redo pipe and all of its problems.

At the same time I though it would be good to separate the existing
message-sending code out of the log-file code.  This is because one
other non-logging message is already supported (the "data" stream in
multiplexed IO) so the addition of two extra messages makes it a
better fit for the io.c code than the log.c code (i.e. when the log
code needs to send a message to another program, it calls the code
in io.c to do it instead of doing it itself).

Finally, the flushing code was tweaked to allow it to signal that
the writing out of the receiver->generator pipe should be completely
flushed (from the in-memory list of messages).  This allows the
final flushing to complete more efficiently than calling io_flush()
in a loop.

In my exuberance for having finally solved the redo-hang problem in
a much better way than my previous suggested fixes for it, I went
ahead and checked in my changes.  However, feedback is still
gratefully accepted.  The changes can be found here:

    http://www.blorf.net/redo.patch

..wayne..