Feature request, or HowTo? State-full resume rsync transfer

[Eberhard Moenkeberg]

Hi,

On Fri, 15 Jul 2011, Donald Pearson wrote:

> I looked in to the -b switch and it's a good idea but I have been unable to
> find a way to use it such that a resume can continue where it left off,
> without re-checking what has already been completed, *and* continue to use
> the same ultimate destination file as a source of diffing.

grep the rsync output for the lines beginning with "backed up" (was 
"keep_backup" in older versions) and take the last line :

cat <rsync stdout> | grep "^backed up" | tail -1 | ...

or have a look into your backup dir - it should contain only one plain 
file under your conditions.

> Looking in to this lead me to the --partial-dir option which really looked
> promising, as it will both use partial transfers to speed up subsequent
> transfers, as well as continue to use the same ultimate desination file as a
> source of diffing, but there's no way to tell rsync not to re-check the
> partial file that exists in the partial-dir.

With the combination of "--b -backup-dir=dir1 --partial-dir=dir2" the 
trick may get very easy.

> --append is incompatible with --partial-dir. :(
>
> If I had a way to instruct rsync to perform a --partial-dir behavior, but do
> not not verify the partial file, that would be exactly what I am looking
> for.

I would try above combination, with a selection between the two partial 
files "by your brain" with a script before each subsequent rsync run 
(after each failing run).

> Matthias,
>
> A vpn tunnel is an interesting idea.  Do you know how long you're able to
> keep rsync in limbo before it will give up?

After reading Matthias' response, that seems to be your best chance to me.

I am using an openvpn tunnel for all my home/workplace connections, and 
this tunnel usually lets survive all tunnelled connections at least for 
several minutes of total physical loss of connection.  After such a loss, 
usually also my home ip number has changed, but the old connections live 
up again.

> I have verified that as long as both the rsync client and server processes
> remain active and the sockets remain open, the physical connection between
> the two can be severed and reconnected and transmission will resume
> upon re-connection.  According to rsync's documentation, when a --timeout
> option is not giving, rsync will not time out natively.
>
> The issue I think is keeping the sockets open, thereby keeping the processes
> active.

Yes, there you go...

Maybe our openvpn "server" point needs to be the rsync server himself if 
he is the point where the network is breaking down - at least you have to 
care that the network connection between the rsync server and your "other 
side" of the tunnel stays unbroken too.


Viele Gruesse
Eberhard Moenkeberg (emoenke at gwdg.de, em at kki.org)

-- 
Eberhard Moenkeberg
Arbeitsgruppe IT-Infrastruktur
E-Mail: emoenke at gwdg.de      Tel.: +49 (0)551 201-1551
-------------------------------------------------------------------------
Gesellschaft fuer wissenschaftliche Datenverarbeitung mbH Goettingen (GWDG)
Am Fassberg 11, 37077 Goettingen
URL:    http://www.gwdg.de             E-Mail: gwdg at gwdg.de
Tel.:   +49 (0)551 201-1510            Fax:    +49 (0)551 201-2150
Geschaeftsfuehrer:         Prof. Dr. Oswald Haan und Dr. Paul Suren
Aufsichtsratsvorsitzender: Prof. Dr. Christian Griesinger
Sitz der Gesellschaft:     Goettingen
Registergericht:           Goettingen  Handelsregister-Nr. B 598
-------------------------------------------------------------------------