Dealing with an unreliable remote

net.rsync at net.rsync at
Tue Nov 25 08:02:52 MST 2014


I've run into a frustrating issue when trying to synchronize a
directory hierarchy over a reliable (but slow) connection to an 
unreliable remote. Basically, I have the following:

This is a set of nested directories containing binaries and sources for
projects I develop/maintain. Every time a new release is made, I deploy
the binaries and sources to an exact copy of the above hierarchy on my
local machine, and then rsync that (over SSH) to

  $ rsync -avz --delete --progress local/

The problem:

The latest project produces one .jar file that's about 80mb.
Evidently, the hosting provider I use for is using some sort
of process tracking system that kills processes that have been running
for too long (I think it just measures CPU time). The result of this is
that I get about 50% of the way through copying that
(comparatively) large file, and then the remote rsync process is
suddenly killed because it has been running for too long.

This would be fine, except that it seems that rsync is utterly refusing
all efforts to continue copying that file from wherever it left off. It
always restarts copying of the file from nothing and tries to copy the
full 80mb, resulting it being killed halfway through and causing much
grinding of teeth.

The documentation for --partial states that "Using the --partial option
tells rsync to keep the partial file which should make a subsequent
transfer of the rest of the file much faster.". Well, for whatever
reason, it doesn't (or it at least fails to continue using it).

I've tried --partial-dir, passing it an absolute path to the temporary
directory in my home directory. It created a file in there the first time, 
but after being killed by the remote side and restarting, it ignored
that file and instead created a new temporary file (with a random suffix) 
in the destination directory! Am I doing something wrong?

  $ rsync -avz --delete --progress --partial-dir=/home/io7m/tmp/rsync

I'm at a loss. How can I reliably get this directory hierarchy up onto
the server? I don't care if I have to retry the command multiple times
until the copy has fully succeeded, but I obviously can't do that if
rsync keeps restarting the failed file from scratch every time.


More information about the rsync mailing list