<div>Eberhard,</div><div><br></div><div>I looked in to the -b switch and it's a good idea but I have been unable to find a way to use it such that a resume can continue where it left off, without re-checking what has already been completed, *and* continue to use the same ultimate destination file as a source of diffing.</div>
<div><br></div><div>Looking in to this lead me to the --partial-dir option which really looked promising, as it will both use partial transfers to speed up subsequent transfers, as well as continue to use the same ultimate desination file as a source of diffing, but there's no way to tell rsync not to re-check the partial file that exists in the partial-dir. </div>
<div><br></div><div>--append is incompatible with --partial-dir. :(</div><div><br></div><div>If I had a way to instruct rsync to perform a --partial-dir behavior, but do not not verify the partial file, that would be exactly what I am looking for.</div>
<div><br></div><div>Brian,</div><div><br></div><div>Yes BT solves many of the delivery problems. I've taken a hard look at BT as an option. The trouble is, as best as I can tell, BT is incapable of diffing like rsync does.</div>
<div><br></div><div>I have considered using BT in combination with a binary diffing utility such as xdelta3, however the patch files that xdelta3 is creating are rather large. With source and desination files ~ 8 gigs in size, the xdelta3 diff patch files are coming out to ~3.5gigs. Better, but rsync can do the job in a little over 1gig, when it is able, so it's hard to sell 3x that traffic. </div>
<div><br></div>Matthias, <div><br></div><div>A vpn tunnel is an interesting idea. Do you know how long you're able to keep rsync in limbo before it will give up?<div><br></div><div>I have verified that as long as both the rsync client and server processes remain active and the sockets remain open, the physical connection between the two can be severed and reconnected and transmission will resume upon re-connection. According to rsync's documentation, when a --timeout option is not giving, rsync will not time out natively.</div>
<div><br></div><div>The issue I think is keeping the sockets open, thereby keeping the processes active.</div><div><br></div><div>When the socket closes I receive an error like this;</div><div><br></div><div><p class="MsoNormal">
sims@SIMS-Usingen:/images/temp$ rsync -zB=512
AllCafeFinal_2011-06-28 <a href="mailto:sims@10.67.2.201:/images/temp/AllCafeFinalTest">sims@10.67.2.201:/images/temp/AllCafeFinalTest</a></p>
<p class="MsoNormal">Read from remote host <a href="http://10.67.2.201">10.67.2.201</a>: Connection reset by peer</p>
<p class="MsoNormal">rsync: writefd_unbuffered failed to write 4 bytes to socket
[sender]: Broken pipe (32)</p>
<p class="MsoNormal">rsync: connection unexpectedly closed (28 bytes received so
far) [sender]</p>
<p class="MsoNormal">rsync error: unexplained error (code 255) at io.c(600)
[sender=3.0.6]</p><div><br></div><div>My first guess was the tcp_fin_timeout setting of the Ubuntu operating system, but this is set for 60 seconds. Leaving everything alone I see roughly 6 minutes before the rsync client errors out.</div>
<div><br></div><div>Knowing that rsync is running over ssh I've experimented with altering the ssh TCPKeepAlive settings on the source(/etc/ssh/ssh_config) and destination (/etc/ssh/sshd_config) to explicitly "no", but nevertheless I will eventually encounter the error. The longest period of time that I've seen the rsync client remain in "limbo" is roughly 16 minutes.</div>
<div><br></div><div>Normaly 16 minutes is an eternity but my end points are connected via satellite and subject to all kinds of strange things such as sand storms, that can knock out coms for hours sometimes.</div><div><br>
</div><div>If truly resuming with rsync is not possible, is it possible to configure my source and destination in such a way that the TCP sessions will never be torn down due to timeouts? Keep in mind there is no NAT between source and destination so there are no middle-man NAT state tables to worry about.</div>
<div><br></div><div>Thanks again for everybody's help and by all means keep the ideas coming. I am trying new angles as I come up with them, I haven't given up on this yet.</div><div><br></div><div>Regards,</div>
<div>Donald<br><br><div class="gmail_quote">On Tue, Jul 12, 2011 at 1:13 PM, Matthias Schniedermeyer <span dir="ltr"><<a href="mailto:ms@citd.de">ms@citd.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
On 12.07.2011 11:10, Donald Pearson wrote:<br>
<br>
...<br>
<br>
A 'trick' i personally use for an unreliable connection is an<br>
OpenSSH-Tunnel.<br>
<br>
Altough any VPN-solution should to the trick.<br>
<br>
That way the connection between the two rsync-halvs isn't directly tied<br>
to the internet-connection.<br>
<br>
In my case that means that when the internet-connection drops the<br>
OpenSSH-Tunnel 'dies' (Assured/Expided by a relative low<br>
'ServerAliveInterval' & 'ClientAliveInterval') but as the rsync<br>
connection isn't directly tied to the internet-conncetion, Linux keeps<br>
that connction 'hanging'. After reconnecting the OpenSSH-Tunnel the<br>
rsync connection resumes when Linux realizes that the destination can be<br>
reached again.<br>
<br>
This also abstracts away problems with Dynamic-IPs.<br>
<div><div></div><div class="h5"><br>
<br>
<br>
<br>
<br>
Bis denn<br>
<br>
--<br>
Real Programmers consider "what you see is what you get" to be just as<br>
bad a concept in Text Editors as it is in women. No, the Real Programmer<br>
wants a "you asked for it, you got it" text editor -- complicated,<br>
cryptic, powerful, unforgiving, dangerous.<br>
<br>
</div></div></blockquote></div><br></div></div></div>