Feature request: Closing destination files in a separate thread

[Kevin Korb]

Can you let rsync do the networking?  If rsync isn't doing the
networking then it isn't much more capable than cp yet it is
significantly slower than cp.

On 12/18/2016 09:13 AM, Adam Nielsen wrote:
> Hi all,
> 
> I'm wondering whether it is feasible to have an option that will make
> rsync spawn a separate thread to close files it has created, to avoid
> the main process blocking while the destination files are flushed during
> the close operation?
> 
> The reason I ask is that it is currently very slow to use rsync on a
> fast locally-mounted network filesystem, as you see the following
> behaviour:
> 
>  1. rsync reads the source file, network remains idle
>  2. Kernel buffers start to fill up
>  3. Some seconds later, kernel starts writing data to destination
>     filesystem
>  4. rsync gets to the end of the file and closes the target file
>  5. rsync hangs for 10+ seconds while the target file's data gets
>     flushed over the network.  No data is being read from the source.
>  6. Back to step 1, rsync reads the next file from the source, while
>     the network is idle as the kernel buffers are now empty.
> 
> This gives the impression that the copy process alternates between
> reading then writing, instead of both reading and writing at the same
> time.  It results in a much slower operation because in step 1 above,
> the network is idle, then in step 5, the local disk is idle.
> 
> I am thinking that if the final close operation was performed in a
> separate thread, it would allow the main rsync operation to continue
> and start copying the next file while the previous one was still being
> flushed.
> 
> This would mean both the source (local disk) and the target (network)
> would be fully utilized instead of them sitting idle for a large amount
> of the operation.
> 
> Is something like this feasible?
> 
> Many thanks,
> Adam.
> 
> P.S. I am using a CIFS mount for this, and when I mount it with
> cache=none then the alternating read-then-write behaviour goes away,
> but the transfer rate drops by almost 30% so it ends up being slower
> overall.
> 
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 191 bytes
Desc: OpenPGP digital signature
URL: <http://lists.samba.org/pipermail/rsync/attachments/20161218/ba7b33ba/signature.sig>