Feature request, or HowTo? State-full resume rsync transfer

[Matthias Schniedermeyer]

On 11.07.2011 16:01, Donald Pearson wrote:
> I am looking to do state-full resume of rsync transfers.
> 
> My network environment is is an unreliable and slow satellite
> infrastructure, and the files I need to send are approaching 10 gigs in
> size.  In this network environment often times links cannot be maintained
> for more than a few minutes at a time.  In this environment, bandwidth is at
> a premium, which is why rsync was chosen as ideal for the job.
> 
> The problem that I am encountering while using rsync in these conditions is
> that the connection between the client and server will drop due to network
> instability before rsync can transfer the entire file.
> 
> Upon retries, rsync starts from the beginning.  Re-checking data that has
> already been sent, as well as re-building the checksum in it's entirety
> every time.  Eventually I reach an impasse where the frequency of link loss
> prevents rsync from ever getting any new data to the destination.
> 
> I've been reading through the various switches in the man pages to try to
> find a combination that will work.  My thinking was to use a combination of
> --partial and --append.  With the first attempt using the --partial switch,
> and subsequent attempts using both --partial and --append.  The idea being
> rsync would build a new "partial" file, and be able to resume building that
> file while making the assumption upon subsequent retries that the existing
> partial file, however large it may be, was assembled correctly and does not
> need to be checked.
> 
> However in practice rsync does not work in this way.

I think you didn't wait for the target rsync to complete, if a 
connection breaks, you have 2 parts left hanging. The less visible 
target-side is the important one here. That rsync has to "complete"
before you do another try. Depending on how your connection drops it MAY 
hang for some time. I don't remember if rsync does "the right thing" 
if you just kill it, or if you have to wait for it. In the latter case 
"--timeout" sounds like it can be used to expedite matters.

And also --inplace, with or without --append, reads like it is what you 
want, if you can live with it's caveats.






Bis denn

-- 
Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.