<br><font size=2 face="sans-serif">I apologise for top posting. The email
client I am forced to use forces me.</font>
<br>
<br><font size=2 face="sans-serif">What Jamie says below seems to imply
to me that the rsync protocol needs to be extended to include clean shutdown.
AFAIK WinSock still behaves the same way it always did so it will not be
able to do the same kind of cleanup that you can in Unix. But Jamie seems
to be saying that on unix a shutdown would cause a RST signal anyway. So
I am really not sure where this leaves me.</font>
<br>
<br><font size=2 face="sans-serif">Surely other people have been seeing
this problem when doing an rsync to hundreds of machines at the same time.
Out of around 200 machines I got these errors from over a dozen today.
That's more than usual but it's all random when it comes down to it. I
wonder who else is rsyncing on this scale.</font>
<br>
<br><font size=2 face="sans-serif">Regards,<br>
<br>
Andrew Marlow<br>
</font>
<br>
<br>
<br>
<table width=100%>
<tr valign=top>
<td width=33%><font size=4 face="helv"><b>Internet </b></font>
<br><font size=1 face="sans-serif"><b>jamie@shareable.org</b></font>
<p><font size=1 face="sans-serif">02/06/2010 13:55</font>
<td width=66%>
<table width=100%>
<tr valign=top>
<td>
<div align=right><font size=1 face="sans-serif">To</font></div>
<td><font size=1 face="sans-serif">Andrew MARLOW</font>
<tr valign=top>
<td>
<div align=right><font size=1 face="sans-serif">cc</font></div>
<td><font size=1 face="sans-serif">rsync@lists.samba.org</font>
<tr valign=top>
<td>
<div align=right><font size=1 face="sans-serif">Subject</font></div>
<td><font size=1 face="sans-serif">Re: rsync 3.0.7 network errors on MS-Windows</font></table>
<br>
<table>
<tr valign=top>
<td>
<td></table>
<br></table>
<br>
<br>
<br><font size=2><tt>andrew.marlow@uk.bnpparibas.com wrote:<br>
> <br>
> I am experiencing intermittent network failures on rsync
3.0.7 built<br>
> using cygwin for Windows-XP (SP2). I am using GCC v4.4.2
and the<br>
> latext version of cygwin.<br>
> The rsync error long indicates things like:<br>
> rsync: writefd_unbuffered failed to write 4092 bytes
to socket<br>
> [generator]:<br>
> Connection reset by peer (104)rsync: read error: Connection
reset by<br>
> peer (104)<br>
> rsync error: error in rsync protocol data stream (code
12) at<br>
> io.c(1530) [generator=3.0.7]<br>
> rsync error: error in rsync protocol data stream (code
12) at<br>
> io.c(760) [receiver=3.0.7]<br>
> Googling I see that these problems were put down to the
way socket are<br>
> cleaned up in Windows and a fix put in place in cleanup.c,
in<br>
> close_all(). But the fix is surrounded by conditional
compilation:-<br>
> #ifdef SHUTDOWN_ALL_SOCKETS<br>
> :<br>
> :<br>
> #endif<br>
> Can someone please explain why that is? Shouldn't the
fix just be<br>
> there always, and regardless of which operating system?<br>
<br>
It's not needed on most operating systems - as the comment there implies.<br>
<br>
According to the notes copied below, SO_LINGER is off by default on<br>
unix sockets, and this means close() will gracefully send the<br>
remaining data in the background, rather than TCP RST. You can assume<br>
that program exit has the same effect as close().<br>
<br>
If SO_LINGER is turned on with a zero timeout, the notes below say TCP<br>
RST is sent on close, which is much like what the comment for<br>
SHUTDOWN_ALL_SOCKETS says is happening on Windows without SO_LINGER.<br>
<br>
Presumably Windows sockets - or at least some version of them (there<br>
are several versions of Winsock) - behaves differently from unix<br>
sockets in this area. It wouldn't be surprising, as historically<br>
Winsock ran inside the process not the kernel, so an exiting process<br>
couldn't implement the unix graceful close behaviour, and maybe they<br>
kept that behaviour the same in later versions.<br>
<br>
That said, I still don't see why SHUTDOWN_ALL_SOCKETS would fix it.<br>
Calling shutdown(fd,2) closes it in both directions, and at least with<br>
usual unix sockets, that would trigger TCP RST anyway if the other end<br>
sends any data after the shutdown.<br>
<br>
Which it seems to be doing: "writefd_unbuffered failed to write 4092<br>
bytes to socket" implies the other end has closed or shutdown(fd,1)
or<br>
shutdown(fd,2), and then data is sent to it which can't be accepted so<br>
the other end sent back TCP RST anyway.<br>
<br>
If rsync is doing that in normal operation, that ought to be a problem<br>
on unix just as much as Windows - and SHUTDOWN_ALL_SOCKETS ought to be<br>
insufficient to prevent the reset.<br>
<br>
Which suggests to me that "writefd_unbuffered failed to write 4092<br>
bytes to socket" is a symptom of a different problem.<br>
<br>
Here are the notes I referred to above.<br>
These are the notes which explain SO_LINGER's behaviour:<br>
<br>
Unix Socket FAQ<br>
http://www.developerweb.net/forum/archive/index.php/t-2982.html<br>
<br>
4.6 - What exactly does SO_LINGER do?<br>
<br>
Contributed by Cyrus Patel<br>
<br>
SO_LINGER affects the behaviour of the close() operation as described<br>
below. No socket operation other than close() is affected by SO_LINGER.<br>
<br>
The following description of the effect of SO_LINGER has been culled<br>
from the setsockopt() and close() man pages for several systems,
but may<br>
still not be applicable to your system. The range of differences
in<br>
implementation ranges from not supporting SO_LINGER at all; or
only<br>
supporting it partially; or having to deal with the "peculiarities"
in a<br>
particular implementation. (see portability notes at end).<br>
<br>
Moreover, the purpose of SO_LINGER is very, very specific and only
a<br>
tiny minority of socket applications actually need it. Unless you
are<br>
extremely familiar with the intricacies of TCP and the BSD socket
API,<br>
you could very easily end up using SO_LINGER in a way for which
it was<br>
not designed.<br>
<br>
<br>
The effect of an setsockopt(..., SO_LINGER,...) depends on what
the<br>
values in the linger structure (the third parameter passed to<br>
setsockopt()) are:<br>
<br>
Case 1: linger->l_onoff is zero (linger->l_linger has no
meaning):<br>
This is the default.<br>
<br>
On close(), the underlying stack attempts to gracefully shutdown
the<br>
connection after ensuring all unsent data is sent. In the case
of<br>
connection-oriented protocols such as TCP, the stack also ensures
that<br>
sent data is acknowledged by the peer. The stack will perform the<br>
above-mentioned graceful shutdown in the background (after the
call to<br>
close() returns), regardless of whether the socket is blocking
or<br>
non-blocking.<br>
<br>
Case 2: linger->l_onoff is non-zero and linger->l_linger
is zero:<br>
<br>
A close() returns immediately. The underlying stack discards any
unsent<br>
data, and, in the case of connection-oriented protocols such as
TCP,<br>
sends a RST (reset) to the peer (this is termed a hard or abortive<br>
close). All subsequent attempts by the peer's application to<br>
read()/recv() data will result in an ECONNRESET.<br>
<br>
Case 3: linger->l_onoff is non-zero and linger->l_linger
is non-zero:<br>
<br>
A close() will either block (if a blocking socket) or fail with<br>
EWOULDBLOCK (if non-blocking) until a graceful shutdown completes
or the<br>
time specified in linger->l_linger elapses (time-out). Upon
time-out the<br>
stack behaves as in case 2 above.<br>
<br>
---------------------------------------------------------------<br>
<br>
Portability note 1: Some implementations of the BSD socket API
do not<br>
implement SO_LINGER at all. On such systems, applying SO_LINGER
either<br>
fails with EINVAL or is (silently) ignored. Having SO_LINGER defined
in<br>
the headers is no guarantee that SO_LINGER is actually implemented.<br>
<br>
Portability note 2: Since the BSD documentation on SO_LINGER is
sparse<br>
and inadequate, it is not surprising to find the various implementations<br>
interpreting the effect of SO_LINGER differently. For instance,
the<br>
effect of SO_LINGER on non-blocking sockets is not mentioned at
all in<br>
BSD documentation, and is consequently treated differently on different<br>
platforms. Taking case 3 for example: Some implementations behave
as<br>
described above. With others, a non-blocking socket close() succeed<br>
immediately leaving the rest to a background process. Others ignore<br>
non-blocking'ness and behave as if the socket were blocking. Yet
others<br>
behave as if SO_LINGER wasn't in effect [as if the case 1, the
default,<br>
was in effect], or ignore linger->l_linger [case 3 is treated
as case<br>
2]. Given the lack of adequate documentation, such differences
are not<br>
(by themselves) indicative of an "incomplete" or "broken"<br>
implementation. They are simply different, not incorrect.<br>
<br>
Portability note 3: Some implementations of the BSD socket API
do not<br>
implement SO_LINGER completely. On such systems, the value of<br>
linger->l_linger is ignored (always treated as if it were zero).<br>
<br>
Technical/Developer note: SO_LINGER does (should) not affect a
stack's<br>
implementation of TIME_WAIT. In any event, SO_LINGER is not the
way to<br>
get around TIME_WAIT. If an application expects to open and close
many<br>
TCP sockets in quick succession, it should be written to use only
a<br>
fixed number and/or range of ports, and apply SO_REUSEPORT to sockets<br>
that use those ports.<br>
<br>
Related note: Many BSD sockets implementations also support a<br>
SO_DONTLINGER socket option. This socket option has the exact opposite<br>
meaning of SO_LINGER, and the two are treated (after inverting
the value<br>
of linger->l_onoff) as equivalent. In other words, SO_LINGER
with a zero<br>
linger->l_onoff is the same as SO_DONTLINGER with a non-zero<br>
linger->l_onoff, and vice versa.<br>
<br>
-- Jamie<br>
</tt></font>
<br><pre>
___________________________________________________________
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is prohibited.
Please refer to http://www.bnpparibas.co.uk/en/information/legal_information.asp?Code=ECAS-845C5H for additional disclosures.
</pre>