Is it possible to transfer a large, dynamic file in a piecemeal fashion?

mail at mail at
Mon Sep 4 22:45:26 UTC 2017

Is it possible to use rsync to transmit a large, dynamic 2TB file in a 
piecemeal fashion during daylight hours over the course of a dozen days?
On a good day, about 200GB of data can be transferred before rsync times
out to enable a nightly local backup to complete. The local backup
changes a few 512 byte blocks in the 2TB file. That's why the 2TB file
is dynamic.

As an experiment, rsync was started on the client to sync the 2TB file.
At that point, rsyncd on the backup server created a temporarily file
with the 2TB file name appended with a dot followed by six characters.

After about 100MB was transferred the rsync client was interrupted and
stopped in order to simulate the nightly shutdown. When rsync was 
restarted it erased the original 100MB temporary file and started over
with a new, empty temporarily file. 

In summary:

* rsync apparently erases temporarily files upon restart

* it takes about a dozen days to transfer the 2TB file

Unless there's an option to have rsync pick up where it left off it
seems impossible to transmit a large file in a piecemeal fashion. 
Because the (nearly) same 200GB at the beginning of the 2TB file gets
retransmitted over and over with each restart.

Thank you,

Don Kuenz KB7RPU

Genius is nothing but a greater aptitude for patience. - de Buffon

More information about the rsync mailing list