Feature request, or HowTo? State-full resume rsync transfer

[Brian K. White]

On 7/15/2011 2:42 PM, Matthias Schniedermeyer wrote:
> On 15.07.2011 13:10, Donald Pearson wrote:
>>
>> Matthias,
>>
>> A vpn tunnel is an interesting idea.  Do you know how long you're able to
>> keep rsync in limbo before it will give up?
>
> I haven't really tried. But it was about 15 Minutes the one time it
> didn't reconnect in time.
>
>> The issue I think is keeping the sockets open, thereby keeping the processes
>> active.
>
> The problem is that TCP really isn't made for transfer over an
> unreliable medium.
>
> Besides rsync, the OSes on both sides also have to keep the connction
> running, but i really don't know which knobs have to be turned to extent
> the time after which the OS times out the connect.
>
>> My first guess was the tcp_fin_timeout setting of the Ubuntu operating
>> system, but this is set for 60 seconds.  Leaving everything alone I see
>> roughly 6 minutes before the rsync client errors out.
>
> tcp_fin is, AFAIK, after you close the socket normaly.
>
>> Thanks again for everybody's help and by all means keep the ideas coming.  I
>> am trying new angles as I come up with them, I haven't given up on this yet.
>
> I got another idea.
>
> If you have the storage on the source-side, to keep the state of the
> target-side. You can create batch-files (--write-batch&  Co.). Of couse
> you also need the storage on the target-side for the batch-files and the
> need space for eventuell growth and temporary files.
>
> You can then copy these batch-files with --append, make sure the
> target-rsync is dead before you retry, for an added bonus i would MD5
> the file on the source side and veryify it after copying. After you have
> transferred a complete batch-file, you can apply it. (Use 'screen' if
> you do it remotely)

That's about what I was trying to suggest myself. Let rsync generate 
delta data locally-only with the batch option, then let bittorrent 
deliver the delta data over the crappy network, then let rsync apply the 
delta locally on the other end.

-- 
bkw