Feature request, or HowTo? State-full resume rsync transfer

Matthias Schniedermeyer ms at citd.de
Tue Jul 12 01:52:00 MDT 2011

On 11.07.2011 16:01, Donald Pearson wrote:
> I am looking to do state-full resume of rsync transfers.
> My network environment is is an unreliable and slow satellite
> infrastructure, and the files I need to send are approaching 10 gigs in
> size.  In this network environment often times links cannot be maintained
> for more than a few minutes at a time.  In this environment, bandwidth is at
> a premium, which is why rsync was chosen as ideal for the job.
> The problem that I am encountering while using rsync in these conditions is
> that the connection between the client and server will drop due to network
> instability before rsync can transfer the entire file.
> Upon retries, rsync starts from the beginning.  Re-checking data that has
> already been sent, as well as re-building the checksum in it's entirety
> every time.  Eventually I reach an impasse where the frequency of link loss
> prevents rsync from ever getting any new data to the destination.
> I've been reading through the various switches in the man pages to try to
> find a combination that will work.  My thinking was to use a combination of
> --partial and --append.  With the first attempt using the --partial switch,
> and subsequent attempts using both --partial and --append.  The idea being
> rsync would build a new "partial" file, and be able to resume building that
> file while making the assumption upon subsequent retries that the existing
> partial file, however large it may be, was assembled correctly and does not
> need to be checked.
> However in practice rsync does not work in this way.

I think you didn't wait for the target rsync to complete, if a 
connection breaks, you have 2 parts left hanging. The less visible 
target-side is the important one here. That rsync has to "complete"
before you do another try. Depending on how your connection drops it MAY 
hang for some time. I don't remember if rsync does "the right thing" 
if you just kill it, or if you have to wait for it. In the latter case 
"--timeout" sounds like it can be used to expedite matters.

And also --inplace, with or without --append, reads like it is what you 
want, if you can live with it's caveats.

Bis denn

Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.

More information about the rsync mailing list