Is it possible to transfer a large, dynamic file in a piecemeal fashion?
Lars Ellenberg
lars.ellenberg at linbit.com
Tue Sep 5 13:23:55 UTC 2017
On Mon, Sep 04, 2017 at 10:45:26PM +0000, Don Kuenz via rsync wrote:
> Greetings,
>
> Is it possible to use rsync to transmit a large, dynamic 2TB file in a
> piecemeal fashion during daylight hours over the course of a dozen days?
> On a good day, about 200GB of data can be transferred before rsync times
> out to enable a nightly local backup to complete. The local backup
> changes a few 512 byte blocks in the 2TB file. That's why the 2TB file
> is dynamic.
>
> As an experiment, rsync was started on the client to sync the 2TB file.
> At that point, rsyncd on the backup server created a temporarily file
> with the 2TB file name appended with a dot followed by six characters.
>
> After about 100MB was transferred the rsync client was interrupted and
> stopped in order to simulate the nightly shutdown. When rsync was
> restarted it erased the original 100MB temporary file and started over
> with a new, empty temporarily file.
>
> In summary:
>
> * rsync apparently erases temporarily files upon restart
>
> * it takes about a dozen days to transfer the 2TB file
>
> Unless there's an option to have rsync pick up where it left off it
> seems impossible to transmit a large file in a piecemeal fashion.
> Because the (nearly) same 200GB at the beginning of the 2TB file gets
> retransmitted over and over with each restart.
Would --partial work for you?
Possibly combine with --partial-dir, --temp-dir,
--delay-updates, and/or --inplace.
--
: Lars Ellenberg
: LINBIT | Keeping the Digital World Running
: DRBD -- Heartbeat -- Corosync -- Pacemaker
: R&D, Integration, Ops, Consulting, Support
More information about the rsync
mailing list