[Bug 5124] Parallelize the rsync run using multiple threads and/or connections
M.Roos at f1-outsourcing.eu
Thu Feb 7 10:47:25 UTC 2019
I made a bash script doing this in parallel, checks how many rsyncs are
running and then starts another 'concurrent one'. My parallel sessions
are against different servers. I doubt if it would make any sense doing
multiple sessions between the same two hosts. My single rsync sessions
was already limited by the hosts iops. So two threads would run at half
IMO rsync does what it needs to do, if you want it to run in parallel
execute it in parallel.
>--- Comment #8 from Michael <michael.williams at infatech.co.nz> ---
>+1 from me on this.
>We have several situations where we need to copy a large number of
>files, and I expect that having multiple file transfer threads,
allowing say ~5
>transfers concurrently, would speed up the process considerably. I
>this would also make better use of the available network bandwidth as
>transfer appears to have an overhead for starting and completing the
So test it with two or 3 concurrent sessions.
>which makes the effective transfer rate far less than the available
>bandwidth. This is the method one of our pieces of backup software
>speed up backups and is also implemented in FileZilla for file
>Consider a very large file that needs to be transferred, along with a
>small files. In a single transfer mode, all other files would need to
>while the large file is transferred. If there are multiple transfers
>concurrently, the smaller files will continue transferring while the
>transfers. I have seen the benefits of this sort of implementation in
>I can also see benefits in having file transfers begin whilst rsync is
>comparing files. This could logically work if you consider rsync makes
>of files to be transferred and that it begins transferring files as
>this list begins to be populated. In situations where there are a
>of files and few of these files changed, the sync could effectively be
>completed by the time rsync is finished comparing files (given the few
>files may have already been transferred during the file comparison).
>is effectively implemented in FileZilla (consider copying a directory
>FileZilla has to recurse into each directory and add each file to copy
>Interestingly, I assumed this was already an option for rsync, so I
>looking to find the necessary option. However, all I found were the
>mentioned hacks, which weren't what I was going for.
More information about the rsync