rsync very very slow with multiple instances at the same time.

[Jayce Piel]

I create a new thread, because the issue is not really the same, but i copy here the thread that made me jump into the list.

My issue is not really that it waits before starting copying, but a general performance issue, specially when there are multiple rsync running at the same time.

Here is my situation :
I have multiple clients (around 20) with users and i want to rsync their home dirs with my server to keep a copy of their local files.
On the server, files are hosted on a iSCSI volume (on a Thecus RAID) where i never had any performance issue before.

When there is only one client, i have no real performance issues. In a few minutes, even with a very large number of files (some users have up to ), the sync is done if there are not too many changed files.
But when there are 3 or more rsync at the same time, all rsync become very very slow and can take a few hours to complete.

Here are my options :

/usr/local/bin/rsync3 --rsync-path=/usr/local/bin/rsync3 -aHXxvE --stats --numeric-ids --delete-excluded --delete-before --human-readable —rsh="ssh -T -c aes128-ctr -o Compression=no -x" -z --skip-compress=gz/bz2/jpg/jpeg/ogg/mp3/mp4/mov/avi/vmdk/vmem --inplace --chmod=u+w --timeout=60 —exclude=‘Caches' —exclude=‘SyncService' —exclude=‘.FileSync' —exclude=‘IMAP*' —exclude=‘.Trash' —exclude='Saved Application State' —exclude='Autosave Information' --exclude-from=/Users/pabittan/.UserSync/exclude-list --max-size=1000M /Users/pabittan/ xserve.local.fftir:./


Here is the version i use (self compiled) : 
$ /usr/local/bin/rsync3 --version
rsync  version 3.1.2-jsp  protocol version 31
Copyright (C) 1996-2015 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes, no prealloc, file-flags

I had to put in place a sort of queue to not allow more than 4 simultaneous rsync to be sure they run at least once a day each. Even limiting to 4 rsync makes some wait hours before starting a backup.

I’m open to any help to improve perfs. (i have put my whole script calling rsync on github : https://github.com/jpiel/UserSync <https://github.com/jpiel/UserSync> )

PS: 
I checked, CPU is not under pressure, each rsync instance use between 2 and 5% CPU. The whole CPU usage 30%.
I also checked network, and it’s not either an issue.
Disk usage doesn’t seem to be at a high load either… (peak at 300 IO/sec)


> Le 20 mars 2018 à 13:00, rsync-request at lists.samba.org a écrit :
> 
> De: Kevin Korb <kmk at sanitarium.net <mailto:kmk at sanitarium.net>>
> Objet: Rép : Very slow to start sync with millions of directories and files
> Date: 19 mars 2018 à 15:33:31 UTC+1
> À: rsync at lists.samba.org <mailto:rsync at lists.samba.org>
> 
> 
> The performance of rsync with a huge number of files is greatly
> determined by every option you are using.  So, what is your whole
> command line?
> 
> On 03/19/2018 09:05 AM, Bráulio Bhavamitra via rsync wrote:
>> Hi all,
>>  
>> I'm using rsync 3 to copy all files from one disk to another. The files
>> were writen by Minio, an S3 compatible opensource backend.
>> 
>> The number of files is dozens of millions, almost each of them within
>> its own directory.
>> 
>> Rsync takes a long time, when not several hours, to even start syncing
>> files. I already see a few reasons:
>> - it first create all directories to put files in, that could be done
>> along with the sync
>> - it needs to generate the list of all files before starting, and cannot
>> start syncing and keep the list generation in a different thread.
>> 
>> Cheers,
>> bráulio
>> 
>> 
> 
> -- 
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> 	Kevin Korb			Phone:    (407) 252-6853
> 	Systems Administrator		Internet:
> 	FutureQuest, Inc.		Kevin at FutureQuest.net <mailto:Kevin at FutureQuest.net>  (work)
> 	Orlando, Florida		kmk at sanitarium.net <mailto:kmk at sanitarium.net> (personal)
> 	Web page:			http://www.sanitarium.net/ <http://www.sanitarium.net/>
> 	PGP public key available on web site.
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-- 
Jayce Piel   —    jayce.piel at gmail.com  --  0616762431
   Responsable Informatique F.F.Tir

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20180321/cf032fa1/attachment.html>