[Bug 13645] New: Improve efficiency when resuming transfer of large files
samba-bugs at samba.org
samba-bugs at samba.org
Fri Oct 5 17:34:31 UTC 2018
https://bugzilla.samba.org/show_bug.cgi?id=13645
Bug ID: 13645
Summary: Improve efficiency when resuming transfer of large
files
Product: rsync
Version: 3.0.9
Hardware: All
OS: All
Status: NEW
Severity: enhancement
Priority: P5
Component: core
Assignee: wayned at samba.org
Reporter: pe1chl at amsat.org
QA Contact: rsync-qa at samba.org
When transferring large files over a slow network, we interrupt rsync at the
beginning of business hours leaving the transfer unfinished.
The command used is: rsync -av --inplace --bwlimit=400 hostname::module /dest
When restarting the transfer, a lot of time is "wasted" while first the local
system is reading the partially transferred file and sends the checksums to the
remote, which only then starts to read the source file until it finds something
to transfer. So nothing happens until 2 times the time required to read the
partial transfer from the disks! When the partial file is many many GB, this
can take hours.
Suggestions:
1. when the source is larger than the destination, immediately begin to
transfer from the offset in the source equal to the size of the destination.
it is already known that this part will have to be transferred.
2. try to do the reading of the partial file at the destination and the same
part of the source in parallel (so the time is halved), and preferably also in
parallel to 1.
Of course these optimizations (at least #2) may actually decrease performance
when the transfer is local (not over slow network) and the disk read rate is
negatively affected by reading at two different places in parallel. So #2
should only be attempted when the transfer is over a network.
--
You are receiving this mail because:
You are the QA Contact for the bug.
More information about the rsync
mailing list