--partial not working?

[Kevin Korb]

Again, --partial only means don't delete the incomplete file if rsync is
aborted.  Normally rsync will delete the incomplete file so you don't
have bogus files laying around.

When you rsync to or from a network mount to rsync that is a local copy.
 To use rsync over the network either your source or your target would
be hostname:/path (for rsync over ssh) or hostname::module (for an
rsyncd server).

With a local copy rsync forces --whole-file because that is a simple
read the file from one place and write it to the other place.  If you
were to override that rsync would have to read both versions of the file
(one of them over the network) compare chunks of the files, the chunks
that are correct would then be written back over the network to a new
file while the chunks that aren't correct would be written over the
network to complete that new file.  This would be significantly more
network transferring.

Adding --checksum would only make things much worse.  Rsync would
checksum every file on both ends (reading them over the network) before
doing exactly what it would have done without --checksum.

Essentially, rsync's differential transfer algorithm is designed to
minimize network IO at the expense of CPU usage and local disk IO.  But
in order to do that you need rsync running on both ends of a network
connection communicating to each other.  That way they can each do
hashing/checksumming on local files quickly and negotiate the smallest
amount of data to send over the network.

On 06/24/2016 03:22 PM, McDowell, Blake wrote:
> Hi Kevin, 
> 
> I'm not a systems manager so my apologies if I'm a little lost here. I'm an audiovisual conservator/archivist and I use rsync for transferring files, a lot. 
> 
> Yes, I connect to the server and then it shows up as a disk on my desktop and I run rsync between the external drive mounted on my computer and the now mounted server. So, this would be a local copy? And, therefore, there is no way to invoke --partial because --whole-file is forced?
> 
> I don't quite understand what you mean by "rsync isn't doing the networking", but if it is not doing this then is there no checksumming occurring during the transfer? Unless, of course, -c.
> 
> Thanks!
> 
> -----Original Message-----
> From: rsync [mailto:rsync-bounces at lists.samba.org] On Behalf Of Kevin Korb
> Sent: Friday, June 24, 2016 3:07 PM
> To: rsync at lists.samba.org
> Subject: Re: --partial not working?
> 
> If you are doing a local only copy (rsync isn't networking) then --whole-file is forced.  There is no benefit of reading and checksumming files to reduce network transfer when there is no network transfer.
> 
> You said you were moving data to a remote server so I assume you are using a network mount of some kind to make it a local copy instead of letting rsync do the networking.
> 
> Anyways, if rsync isn't doing the networking then forcing it to do a delta transfer would only make it take even longer.
> 
> On 06/24/2016 02:59 PM, McDowell, Blake wrote:
>> Hi Kevin,
>>
>> I haven't specified --whole-file. After entering an rsync command the terminal always reads "delta-transmission disabled for local transfer or --whole-file" but I assume that is just a standard phrase that always appears. 
>>
>> So, if I am running partial (-P) and not using --whole-file or disabling the delta-transmission, why would an incomplete file be deleted and the transfer start anew after an interruption?
>>
>> Blake
>>
>> -----Original Message-----
>> From: rsync [mailto:rsync-bounces at lists.samba.org] On Behalf Of Kevin 
>> Korb
>> Sent: Friday, June 24, 2016 12:03 PM
>> To: rsync at lists.samba.org
>> Subject: Re: --partial not working?
>>
>> Partial means don't delete the incomplete file.  The file is being transferred because it doesn't match.  But rsync does delta-transfers so it won't actually transfer the parts that match.  It will just verify that they do match.
>>
>> Unless of course --whole-file.
>>
>> On 06/24/2016 11:47 AM, McDowell, Blake wrote:
>>> Hello,
>>>
>>>  
>>>
>>> I’m running rsync -avPhi to move large video files to a remote server.
>>>  Often we have to stop a transfer midway through to push something 
>>> else to the server. My hope was that the -P flag would invoke 
>>> --partial and the transfer would pick-up where it left off. This does 
>>> not seem to be happening. Rather, the transfer starts over from the 
>>> beginning, erasing the partially transferred file.
>>>
>>>
>>> When I redo the transfer my I see this:
>>>
>>>> f.st..g.... afc2010039_crhp0026_mv06_concat_Williams_Cecil_J.mov
>>>
>>>
>>> I understand what all of that means, but I don't know why it doesn't 
>>> resume the partial transfer.
>>>
>>>
>>> Do I need to change my rsync command to allow it to resume a transfer 
>>> of a partially transferred file?
>>>
>>>
>>> Thanks,
>>>
>>> Blake
>>>
>>>
>>>
>>
>> --
>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>> 	Kevin Korb			Phone:    (407) 252-6853
>> 	Systems Administrator		Internet:
>> 	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
>> 	Orlando, Florida		kmk at sanitarium.net (personal)
>> 	Web page:			http://www.sanitarium.net/
>> 	PGP public key available on web site.
>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._
>> .,
>>
> 
> --
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> 	Kevin Korb			Phone:    (407) 252-6853
> 	Systems Administrator		Internet:
> 	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
> 	Orlando, Florida		kmk at sanitarium.net (personal)
> 	Web page:			http://www.sanitarium.net/
> 	PGP public key available on web site.
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.samba.org/pipermail/rsync/attachments/20160624/6b6b0b0e/signature.sig>