Is it possible to transfer a large, dynamic file in a piecemeal fashion? Yes it is.

[Kevin Korb]

You probably want --inplace for this rather than partial.  The upside of
--partial is that the existing file isn't replaced until there is a
complete file to replace it with.  The downside is building up that
entire new file.

On 09/05/2017 12:01 AM, Don Kuenz via rsync wrote:
> Subject: Re: Is it possible to transfer a large, dynamic file in a piecemeal fashion? Yes it is.
> 
> Don Kuenz via rsync <rsync at lists.samba.org> wrote:
>> Greetings,
>>
>> Is it possible to use rsync to transmit a large, dynamic 2TB file in a
>> piecemeal fashion during daylight hours over the course of a dozen days?
>> On a good day, about 200GB of data can be transferred before rsync times
>> out to enable a nightly local backup to complete. The local backup
>> changes a few 512 byte blocks in the 2TB file. That's why the 2TB file
>> is dynamic.
>>
>> As an experiment, rsync was started on the client to sync the 2TB file.
>> At that point, rsyncd on the backup server created a temporarily file
>> with the 2TB file name appended with a dot followed by six characters.
>>
>> After about 100MB was transferred the rsync client was interrupted and
>> stopped in order to simulate the nightly shutdown. When rsync was
>> restarted it erased the original 100MB temporary file and started over
>> with a new, empty temporarily file.
>>
>> In summary:
>>
>> * rsync apparently erases temporarily files upon restart
>>
>> * it takes about a dozen days to transfer the 2TB file
>>
>> Unless there's an option to have rsync pick up where it left off it
>> seems impossible to transmit a large file in a piecemeal fashion.
>> Because the (nearly) same 200GB at the beginning of the 2TB file gets
>> retransmitted over and over with each restart.
> 
> The rsync --partial argument retains the partial file between an 
> interruption and a restart. It works for --timelimit interruptions as
> well as ctrl-c interruptions on the client side.    
> 
> Thank you,
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 224 bytes
Desc: OpenPGP digital signature
URL: <http://lists.samba.org/pipermail/rsync/attachments/20170904/c88e1b5b/signature.sig>