Feature request, or HowTo? State-full resume rsync transfer
Brian K. White
brian at aljex.com
Fri Jul 15 14:26:16 MDT 2011
On 7/15/2011 2:42 PM, Matthias Schniedermeyer wrote:
> On 15.07.2011 13:10, Donald Pearson wrote:
>> A vpn tunnel is an interesting idea. Do you know how long you're able to
>> keep rsync in limbo before it will give up?
> I haven't really tried. But it was about 15 Minutes the one time it
> didn't reconnect in time.
>> The issue I think is keeping the sockets open, thereby keeping the processes
> The problem is that TCP really isn't made for transfer over an
> unreliable medium.
> Besides rsync, the OSes on both sides also have to keep the connction
> running, but i really don't know which knobs have to be turned to extent
> the time after which the OS times out the connect.
>> My first guess was the tcp_fin_timeout setting of the Ubuntu operating
>> system, but this is set for 60 seconds. Leaving everything alone I see
>> roughly 6 minutes before the rsync client errors out.
> tcp_fin is, AFAIK, after you close the socket normaly.
>> Thanks again for everybody's help and by all means keep the ideas coming. I
>> am trying new angles as I come up with them, I haven't given up on this yet.
> I got another idea.
> If you have the storage on the source-side, to keep the state of the
> target-side. You can create batch-files (--write-batch& Co.). Of couse
> you also need the storage on the target-side for the batch-files and the
> need space for eventuell growth and temporary files.
> You can then copy these batch-files with --append, make sure the
> target-rsync is dead before you retry, for an added bonus i would MD5
> the file on the source side and veryify it after copying. After you have
> transferred a complete batch-file, you can apply it. (Use 'screen' if
> you do it remotely)
That's about what I was trying to suggest myself. Let rsync generate
delta data locally-only with the batch option, then let bittorrent
deliver the delta data over the crappy network, then let rsync apply the
delta locally on the other end.
More information about the rsync