Feature request, or HowTo? State-full resume rsync transfer

[Donald Pearson]

Wow can't tell you how many times I've read over the man page but Batch Mode
never jumped out at me before.

Batch Mode could very well provide a solution, I'm going to give it a run.

Also, I think I found the knob in Ubuntu that's creating the error in rsync.
 It's /proc/sys/net/ipv4/tcp_retries2  which governs the number of times to
attempt to re-transmit a TCP packet.  The default value is 15 which provided
about a 16 minute window to failure.  Turning it down to 5 reduced the
window to 20 seconds, where turning it up a little to 8 increased the window
to 2 minutes, so it's definitely on a curve.

I think with a combination of tuning tcp_retries2 and rsync Batch Mode I may
be able to pull together a solution.

Regards,
Donald

On Fri, Jul 15, 2011 at 4:26 PM, Brian K. White <brian at aljex.com> wrote:

> On 7/15/2011 2:42 PM, Matthias Schniedermeyer wrote:
>
>> On 15.07.2011 13:10, Donald Pearson wrote:
>>
>>>
>>> Matthias,
>>>
>>> A vpn tunnel is an interesting idea.  Do you know how long you're able to
>>> keep rsync in limbo before it will give up?
>>>
>>
>> I haven't really tried. But it was about 15 Minutes the one time it
>> didn't reconnect in time.
>>
>>  The issue I think is keeping the sockets open, thereby keeping the
>>> processes
>>> active.
>>>
>>
>> The problem is that TCP really isn't made for transfer over an
>> unreliable medium.
>>
>> Besides rsync, the OSes on both sides also have to keep the connction
>> running, but i really don't know which knobs have to be turned to extent
>> the time after which the OS times out the connect.
>>
>>  My first guess was the tcp_fin_timeout setting of the Ubuntu operating
>>> system, but this is set for 60 seconds.  Leaving everything alone I see
>>> roughly 6 minutes before the rsync client errors out.
>>>
>>
>> tcp_fin is, AFAIK, after you close the socket normaly.
>>
>>  Thanks again for everybody's help and by all means keep the ideas coming.
>>>  I
>>> am trying new angles as I come up with them, I haven't given up on this
>>> yet.
>>>
>>
>> I got another idea.
>>
>> If you have the storage on the source-side, to keep the state of the
>> target-side. You can create batch-files (--write-batch&  Co.). Of couse
>> you also need the storage on the target-side for the batch-files and the
>> need space for eventuell growth and temporary files.
>>
>> You can then copy these batch-files with --append, make sure the
>> target-rsync is dead before you retry, for an added bonus i would MD5
>> the file on the source side and veryify it after copying. After you have
>> transferred a complete batch-file, you can apply it. (Use 'screen' if
>> you do it remotely)
>>
>
> That's about what I was trying to suggest myself. Let rsync generate delta
> data locally-only with the batch option, then let bittorrent deliver the
> delta data over the crappy network, then let rsync apply the delta locally
> on the other end.
>
> --
> bkw
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: https://lists.samba.org/**
> mailman/listinfo/rsync <https://lists.samba.org/mailman/listinfo/rsync>
> Before posting, read: http://www.catb.org/~esr/faqs/**smart-questions.html<http://www.catb.org/~esr/faqs/smart-questions.html>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20110715/7e275ef3/attachment.html>