Feature request, or HowTo? State-full resume rsync transfer
donaldwhpearson at gmail.com
Fri Jul 15 14:59:46 MDT 2011
Wow can't tell you how many times I've read over the man page but Batch Mode
never jumped out at me before.
Batch Mode could very well provide a solution, I'm going to give it a run.
Also, I think I found the knob in Ubuntu that's creating the error in rsync.
It's /proc/sys/net/ipv4/tcp_retries2 which governs the number of times to
attempt to re-transmit a TCP packet. The default value is 15 which provided
about a 16 minute window to failure. Turning it down to 5 reduced the
window to 20 seconds, where turning it up a little to 8 increased the window
to 2 minutes, so it's definitely on a curve.
I think with a combination of tuning tcp_retries2 and rsync Batch Mode I may
be able to pull together a solution.
On Fri, Jul 15, 2011 at 4:26 PM, Brian K. White <brian at aljex.com> wrote:
> On 7/15/2011 2:42 PM, Matthias Schniedermeyer wrote:
>> On 15.07.2011 13:10, Donald Pearson wrote:
>>> A vpn tunnel is an interesting idea. Do you know how long you're able to
>>> keep rsync in limbo before it will give up?
>> I haven't really tried. But it was about 15 Minutes the one time it
>> didn't reconnect in time.
>> The issue I think is keeping the sockets open, thereby keeping the
>> The problem is that TCP really isn't made for transfer over an
>> unreliable medium.
>> Besides rsync, the OSes on both sides also have to keep the connction
>> running, but i really don't know which knobs have to be turned to extent
>> the time after which the OS times out the connect.
>> My first guess was the tcp_fin_timeout setting of the Ubuntu operating
>>> system, but this is set for 60 seconds. Leaving everything alone I see
>>> roughly 6 minutes before the rsync client errors out.
>> tcp_fin is, AFAIK, after you close the socket normaly.
>> Thanks again for everybody's help and by all means keep the ideas coming.
>>> am trying new angles as I come up with them, I haven't given up on this
>> I got another idea.
>> If you have the storage on the source-side, to keep the state of the
>> target-side. You can create batch-files (--write-batch& Co.). Of couse
>> you also need the storage on the target-side for the batch-files and the
>> need space for eventuell growth and temporary files.
>> You can then copy these batch-files with --append, make sure the
>> target-rsync is dead before you retry, for an added bonus i would MD5
>> the file on the source side and veryify it after copying. After you have
>> transferred a complete batch-file, you can apply it. (Use 'screen' if
>> you do it remotely)
> That's about what I was trying to suggest myself. Let rsync generate delta
> data locally-only with the batch option, then let bittorrent deliver the
> delta data over the crappy network, then let rsync apply the delta locally
> on the other end.
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: https://lists.samba.org/**
> mailman/listinfo/rsync <https://lists.samba.org/mailman/listinfo/rsync>
> Before posting, read: http://www.catb.org/~esr/faqs/**smart-questions.html<http://www.catb.org/~esr/faqs/smart-questions.html>
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the rsync