Comparison of huge files ineffective?

[Kevin Korb]

Your observation would be right if you are using --checksum which you
shouldn't be.  Otherwise, unless you are using --whole-file rsync will
use its differential algorithm to compare the files.  If you are using
--progress you will see it step through the file at a faster speed than
the network is capable of.  If there is no difference then rsync will go
very fast to 100% then do any metadata changes.  If it hits a difference
and you aren't using --inplace then rsync will copy the correct
beginning of the file to a new temp file then begin transferring the
differences.  On huge files this can cause a significant pause while it
does that local copy to a temp file.

On 03/14/2018 09:25 AM, Heiko Schlittermann via rsync wrote:
> Hi,
> 
> 
> how does rsync work if it compares two very huge files on two distinct
> hosts (rsync uses a networked connection, via SSH)?
> 
> Some observation seems to indicate, that rsync first reads (and
> checksums?) the remote (destination) side, then, if finished, it reads
> (and checksums?) the local (source) side and then starts transferring
> the delta if done with the comparison of these two files on both sides.
> 
> The major problem (with respect to runtime) seems to be the missing
> parellism in read (and checksumming) the source and the destination.
> 
> 1)  Is this observation right?
> 1a) Does it depend on the rsync version?
> 
> 2)  Is there a way to get read (and checksumming) of both sides
>     in parallel?
> 
> 
>     Best regards from Dresden/Germany
>     Viele Grüße aus Dresden
>     Heiko Schlittermann
> 
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 224 bytes
Desc: OpenPGP digital signature
URL: <http://lists.samba.org/pipermail/rsync/attachments/20180314/10729e7c/signature.sig>