Feature request, or HowTo? State-full resume rsync transfer
ms at citd.de
Tue Jul 12 01:52:00 MDT 2011
On 11.07.2011 16:01, Donald Pearson wrote:
> I am looking to do state-full resume of rsync transfers.
> My network environment is is an unreliable and slow satellite
> infrastructure, and the files I need to send are approaching 10 gigs in
> size. In this network environment often times links cannot be maintained
> for more than a few minutes at a time. In this environment, bandwidth is at
> a premium, which is why rsync was chosen as ideal for the job.
> The problem that I am encountering while using rsync in these conditions is
> that the connection between the client and server will drop due to network
> instability before rsync can transfer the entire file.
> Upon retries, rsync starts from the beginning. Re-checking data that has
> already been sent, as well as re-building the checksum in it's entirety
> every time. Eventually I reach an impasse where the frequency of link loss
> prevents rsync from ever getting any new data to the destination.
> I've been reading through the various switches in the man pages to try to
> find a combination that will work. My thinking was to use a combination of
> --partial and --append. With the first attempt using the --partial switch,
> and subsequent attempts using both --partial and --append. The idea being
> rsync would build a new "partial" file, and be able to resume building that
> file while making the assumption upon subsequent retries that the existing
> partial file, however large it may be, was assembled correctly and does not
> need to be checked.
> However in practice rsync does not work in this way.
I think you didn't wait for the target rsync to complete, if a
connection breaks, you have 2 parts left hanging. The less visible
target-side is the important one here. That rsync has to "complete"
before you do another try. Depending on how your connection drops it MAY
hang for some time. I don't remember if rsync does "the right thing"
if you just kill it, or if you have to wait for it. In the latter case
"--timeout" sounds like it can be used to expedite matters.
And also --inplace, with or without --append, reads like it is what you
want, if you can live with it's caveats.
Real Programmers consider "what you see is what you get" to be just as
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated,
cryptic, powerful, unforgiving, dangerous.
More information about the rsync