[Bug 13645] New: Improve efficiency when resuming transfer of large files

L A Walsh rsync at tlinx.org
Fri Oct 12 22:15:14 UTC 2018


If you are doing a local<-> local transfer, you are wasting time
with checksums.  You'll get faster performance with "--whole-file".

Why do you stop it at night when you could 'unlimit' the transfer speed?
Seems like when you aren't there would be best time to copy everything.

Doing checksums will cause a noticeable impact to local-file transfers.


On 10/5/2018 10:34 AM, just subscribed for rsync-qa from bugzilla via 
rsync wrote:
> https://bugzilla.samba.org/show_bug.cgi?id=13645
> When transferring large files over a slow network, ...
> The command used is: rsync -av --inplace --bwlimit=400 hostname::module /dest
>
> When restarting the transfer, a lot of time is "wasted" while first the local
> system is reading the partially transferred file and sends the checksums to the remote, ...
>
>   
> Of course these optimizations (at least #2) may actually decrease performance
> when the transfer is local (not over slow network) and the disk read rate is
> negatively affected by reading at two different places in parallel.  So #2
> should only be attempted when the transfer is over a network.
>   
---
    Or might decrease performance on a fast network.  Not sure what you mean
by 'slow' 10Mb?  100Mb -- not sure w/o measuring if it is faster or 
slower to
do checksums, but I know at 1000Mb and 10Gb, checksums are prohibitively
expensive.

NOTE: you also might look at the protocol you use to do network transfers.
I.e. use rsync over a locally mounted disk to a locally mounted network 
share,
and make the network share a samba one.  That way you will get parallelism
automatically -- the file transfer cpu-time will happen inside of samba,
while the local file gathering will happen in rsync.

I regularly got ~ 119MB R/W over 1000Mb ethernet.  BTW, Any place I use a
power-of-2 unit like 'B' (Byte), I use the power-of-two base (1024) prefix,
but if I use a singular unit like 'b' (bit), then I use decimal prefixes.
Doing otherwise makes things hard to calculate and can introduce calculation
inaccuracies.




More information about the rsync mailing list