Speeding up rsync

[hancooper]

Sent with ProtonMail Secure Email.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Tuesday, August 24, 2021 9:40 PM, Kevin Korb <kmk at sanitarium.net> wrote:

> Most of the speedup is due to ssh being bound to a single CPU. You will
> get a little more speed out of a single CPU system but the speedup is
> unlikely to go past 2 in parallel. Even with that you may end up adding
> more latency via disk seeks to accomplish much.


I could run multiple instances of rsync for different parts of the transfer,
and throw them to multiple backup hosts.



> On 8/24/21 5:12 PM, hancooper wrote:
>
> > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > On Tuesday, August 24, 2021 8:25 PM, Kevin Korb via rsync rsync at lists.samba.org wrote:
> >
> > > In my experience backing up multiple hosts to a single host can speed
> > > things up. Especially if using rsync over ssh with multiple CPUs on the
> > > single host. You would need to do some experimentation to determine the
> > > best number for your hardware and network. Also, if you exceed that
> > > number you can put more stress on the backup machine lessening the
> > > impact to the hosts being backed up.
> > > On 8/24/21 3:58 PM, hancooper via rsync wrote:
> > >
> > > > It makes sense to we to run multiple rsync commands to speed up a backup.
> > > > At work, some have argued that if I sync the data all to the same host,
> > > > there is no advantage
> > > > in parallelization.   Whether you sync 3x 1G in parallel or 1x 3G makes
> > > > no noticeable difference.
> > > > Speed in general does not increase by increasing the number of sync
> > > > processes.
> > > > However, if one want to sync to multiple hosts, the effort may be
> > > > worthwhile.
> > > > What is the current advice about speeding up data transfer ?
> > >
> > > --
> > > Kevin Korb Phone: (407) 252-6853
> >
> > Hi Kevin. What about a single host to another with multiple CPU's ?
>
> --
>
> ~-,._.,-~'`^`'~-,._.,-~'`^`'~-,._.,-~'`^`'~-,._.,-~'`^`'~-,._.,
> Kevin Korb Phone: (407) 252-6853
> Systems Administrator Internet:
> FutureQuest, Inc. Kevin at FutureQuest.net (work)
> Orlando, Florida kmk at sanitarium.net (personal)
> Web page: https://sanitarium.net/
> PGP public key available on web site.
> ~-,..,-~'`^`'~-,..,-~'`^`'~-,..,-~'`^`'~-,..,-~'`^`'~-,._.,