superlifter design notes and a new proposal

[Wayne Davison]

On Tue, 6 Aug 2002, Martin Pool wrote:
> What are your thoughts about error reporting?  Are they going to be
> messages back in the other direction?

I've been wondering about this myself.  My first rZync implementation
had the remote side put the strings into the regular message datastream
so that it got back to the user's side (where it was output).  The only
complicating factor seems to be fatal errors:  you can't just queue up
the error message and die, since that would cause the pipe to break and
the other side would likely never see the error.  I put some special
code into rZync that attempts to keep running in "aborting" mode, which
means that it keeps reading the input data (since the remote process
might block if we stopped reading) while continuing to flush any
remaining output data.  Most input messages are discarded in this mode,
but we continue to look for a message that indicates that the other side
successfully got our fatal error message before we allow ourselves to
die.

This weird "aborting" mode has been difficult to get to work reliably,
though, so I switched over to trying a different error-reporting
technique.  I currently have the remote process output the error strings
(and warnings and verbose output) on stderr, which lets the remote shell
send the error back over a parallel data path.  The nice thing about
this is that we don't have to process any of the regular I/O in order
for the output to reach the other side.  I've found this to be more
reliable in getting fatal error messages to be returned from the remote
process, but I'm curious if there's some gotcha that might make this
method undesirable (for instance, if some remote-shell programs didn't
support a separate stdout/stderr data path).

If we end up sending human-readable output strings via stderr, I think
we will still need to have a message that can be sent over the normal
data path to indicate to the other side that something has gone wrong in
the transfer.  If there wasn't a "we must die now" fatal error
condition, we will want to make a note of the problem (e.g. for the exit
value) and possibly make a controlled exit.  I think the best way to
convey this to the other side is through a normal message.

The final complicating factor is if we support a mode where the transfer
program is controlled by another program -- how does this program get
reliable error reporting back?  I haven't thought about this area too
much yet.

What do you think?

..wayne..