<html><head></head><body><div style="font-family: Verdana;font-size: 12.0px;"><div>
<div>most likely, you ovestrain your NAS with disk random IOPs. furthermore, iSCSI is an additional throttle here, making things worse.</div>
<div> </div>
<div>your issue is probably centered around metadata reads/latency...</div>
<div> </div>
<div>have a look on IO-Wait on the server/nas side...</div>
<div> </div>
<div>regards</div>
<div>roland</div>
<div> </div>
<div>
<div name="quote" style="margin:10px 5px 5px 10px; padding: 10px 0 10px 10px; border-left:2px solid #C3D9E5; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div style="margin:0 0 10px 0;"><b>Gesendet:</b> Mittwoch, 21. März 2018 um 14:49 Uhr<br/>
<b>Von:</b> "Jayce Piel via rsync" <rsync@lists.samba.org><br/>
<b>An:</b> rsync@lists.samba.org<br/>
<b>Betreff:</b> rsync very very slow with multiple instances at the same time.</div>
<div name="quoted-content">
<div>
<div>I create a new thread, because the issue is not really the same, but i copy here the thread that made me jump into the list.</div>
<div> </div>
<div>My issue is not really that it waits before starting copying, but a general performance issue, specially when there are multiple rsync running at the same time.</div>
<div> </div>
<div>Here is my situation :</div>
<div>I have multiple clients (around 20) with users and i want to rsync their home dirs with my server to keep a copy of their local files.</div>
<div>On the server, files are hosted on a iSCSI volume (on a Thecus RAID) where i never had any performance issue before.</div>
<div> </div>
<div>When there is only one client, i have no real performance issues. In a few minutes, even with a very large number of files (some users have up to ), the sync is done if there are not too many changed files.</div>
<div>But when there are 3 or more rsync at the same time, all rsync become very very slow and can take a few hours to complete.</div>
<div> </div>
<div>Here are my options :</div>
<div> </div>
<div>/usr/local/bin/rsync3 --rsync-path=/usr/local/bin/rsync3 -aHXxvE --stats --numeric-ids --delete-excluded --delete-before --human-readable —rsh="ssh -T -c aes128-ctr -o Compression=no -x" -z --skip-compress=gz/bz2/jpg/jpeg/ogg/mp3/mp4/mov/avi/vmdk/vmem --inplace --chmod=u+w --timeout=60 —exclude=‘Caches' —exclude=‘SyncService' —exclude=‘.FileSync' —exclude=‘IMAP*' —exclude=‘.Trash' —exclude='Saved Application State' —exclude='Autosave Information' --exclude-from=/Users/pabittan/.UserSync/exclude-list --max-size=1000M /Users/pabittan/ xserve.local.fftir:./</div>
<div>
<div> </div>
<div>Here is the version i use (self compiled) : </div>
<div>
<div>$ /usr/local/bin/rsync3 --version</div>
<div>rsync version 3.1.2-jsp protocol version 31</div>
<div>Copyright (C) 1996-2015 by Andrew Tridgell, Wayne Davison, and others.</div>
<div>Web site: <a href="http://rsync.samba.org/" target="_blank">http://rsync.samba.org/</a></div>
<div>Capabilities:</div>
<div> 64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,</div>
<div> socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,</div>
<div> append, ACLs, xattrs, iconv, symtimes, no prealloc, file-flags</div>
<div> </div>
<div>I had to put in place a sort of queue to not allow more than 4 simultaneous rsync to be sure they run at least once a day each. Even limiting to 4 rsync makes some wait hours before starting a backup.</div>
<div> </div>
<div>I’m open to any help to improve perfs. (i have put my whole script calling rsync on github : <a href="https://github.com/jpiel/UserSync" target="_blank">https://github.com/jpiel/UserSync</a> )</div>
<div> </div>
<div>PS: </div>
<div>I checked, CPU is not under pressure, each rsync instance use between 2 and 5% CPU. The whole CPU usage 30%.</div>
<div>I also checked network, and it’s not either an issue.</div>
<div>Disk usage doesn’t seem to be at a high load either… (peak at 300 IO/sec)</div>
<div> </div>
</div>
<div>
<blockquote>
<div>Le 20 mars 2018 à 13:00, <a href="mailto:rsync-request@lists.samba.org" onclick="parent.window.location.href='rsync-request@lists.samba.org'; return false;" target="_blank">rsync-request@lists.samba.org</a> a écrit :</div>
<div>
<div style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;margin: 0.0px;"><span style="font-family: system-font , "Helvetica Neue" , Helvetica , sans-serif;color: rgb(127,127,127);"><b>De:<span class="Apple-converted-space"> </span></b></span><span style="font-family: system-font , "Helvetica Neue" , Helvetica , sans-serif;">Kevin Korb <<a href="mailto:kmk@sanitarium.net" onclick="parent.window.location.href='kmk@sanitarium.net'; return false;" target="_blank">kmk@sanitarium.net</a>></span></div>
<div style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;margin: 0.0px;"><span style="font-family: system-font , "Helvetica Neue" , Helvetica , sans-serif;color: rgb(127,127,127);"><b>Objet:<span class="Apple-converted-space"> </span></b></span><span style="font-family: system-font , "Helvetica Neue" , Helvetica , sans-serif;"><b>Rép : Very slow to start sync with millions of directories and files</b></span></div>
<div style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;margin: 0.0px;"><span style="font-family: system-font , "Helvetica Neue" , Helvetica , sans-serif;color: rgb(127,127,127);"><b>Date:<span class="Apple-converted-space"> </span></b></span><span style="font-family: system-font , "Helvetica Neue" , Helvetica , sans-serif;">19 mars 2018 à 15:33:31 UTC+1</span></div>
<div style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;margin: 0.0px;"><span style="font-family: system-font , "Helvetica Neue" , Helvetica , sans-serif;color: rgb(127,127,127);"><b>À:<span class="Apple-converted-space"> </span></b></span><span style="font-family: system-font , "Helvetica Neue" , Helvetica , sans-serif;"><a href="mailto:rsync@lists.samba.org" onclick="parent.window.location.href='rsync@lists.samba.org'; return false;" target="_blank">rsync@lists.samba.org</a></span></div>
<br style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;"/>
<br style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;"/>
<span style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;float: none;display: inline;">The performance of rsync with a huge number of files is greatly</span><br style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;"/>
<span style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;float: none;display: inline;">determined by every option you are using. So, what is your whole</span><br style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;"/>
<span style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;float: none;display: inline;">command line?</span><br style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;"/>
<br style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;"/>
<span style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;float: none;display: inline;">On 03/19/2018 09:05 AM, Bráulio Bhavamitra via rsync wrote:</span>
<blockquote style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;">Hi all,<br/>
<br/>
I'm using rsync 3 to copy all files from one disk to another. The files<br/>
were writen by Minio, an S3 compatible opensource backend.<br/>
<br/>
The number of files is dozens of millions, almost each of them within<br/>
its own directory.<br/>
<br/>
Rsync takes a long time, when not several hours, to even start syncing<br/>
files. I already see a few reasons:<br/>
- it first create all directories to put files in, that could be done<br/>
along with the sync<br/>
- it needs to generate the list of all files before starting, and cannot<br/>
start syncing and keep the list generation in a different thread.<br/>
<br/>
Cheers,<br/>
bráulio<br/>
<br/>
</blockquote>
<br style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;"/>
<span style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;float: none;display: inline;">--<span class="Apple-converted-space"> </span></span><br style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;"/>
<span style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;float: none;display: inline;">~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,</span><br style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;"/>
<span style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;float: none;display: inline;">Kevin Korb</span><span class="Apple-tab-span" style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: pre;word-spacing: 0.0px;text-decoration: none;"> </span><span class="Apple-tab-span" style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: pre;word-spacing: 0.0px;text-decoration: none;"> </span><span class="Apple-tab-span" style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: pre;word-spacing: 0.0px;text-decoration: none;"> </span><span style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;float: none;display: inline;">Phone: (407) 252-6853</span><br style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;"/>
<span style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;float: none;display: inline;">Systems Administrator</span><span class="Apple-tab-span" style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: pre;word-spacing: 0.0px;text-decoration: none;"> </span><span class="Apple-tab-span" style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: pre;word-spacing: 0.0px;text-decoration: none;"> </span><span style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;float: none;display: inline;">Internet:</span><br style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;"/>
<span style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;float: none;display: inline;">FutureQuest, Inc.</span><span class="Apple-tab-span" style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: pre;word-spacing: 0.0px;text-decoration: none;"> </span><span class="Apple-tab-span" style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: pre;word-spacing: 0.0px;text-decoration: none;"> </span><a href="mailto:Kevin@FutureQuest.net" onclick="parent.window.location.href='Kevin@FutureQuest.net'; return false;" style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;" target="_blank">Kevin@FutureQuest.net</a><span style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;float: none;display: inline;"><span class="Apple-converted-space"> </span> (work)</span><br style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;"/>
<span style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;float: none;display: inline;">Orlando, Florida</span><span class="Apple-tab-span" style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: pre;word-spacing: 0.0px;text-decoration: none;"> </span><span class="Apple-tab-span" style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: pre;word-spacing: 0.0px;text-decoration: none;"> </span><a href="mailto:kmk@sanitarium.net" onclick="parent.window.location.href='kmk@sanitarium.net'; return false;" style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;" target="_blank">kmk@sanitarium.net</a><span style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;float: none;display: inline;"><span class="Apple-converted-space"> </span>(personal)</span><br style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;"/>
<span style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;float: none;display: inline;">Web page:</span><span class="Apple-tab-span" style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: pre;word-spacing: 0.0px;text-decoration: none;"> </span><span class="Apple-tab-span" style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: pre;word-spacing: 0.0px;text-decoration: none;"> </span><span class="Apple-tab-span" style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: pre;word-spacing: 0.0px;text-decoration: none;"> </span><a href="http://www.sanitarium.net/" style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;" target="_blank">http://www.sanitarium.net/</a><br style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;"/>
<span style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;float: none;display: inline;">PGP public key available on web site.</span><br style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;"/>
<span style="font-family: TimesNewRomanPSMT;font-size: 14.0px;font-style: normal;font-weight: normal;letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;text-decoration: none;float: none;display: inline;">~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,</span></div>
</blockquote>
</div>
<div>
<div style="color: rgb(0,0,0);letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;">
<div style="color: rgb(0,0,0);letter-spacing: normal;text-indent: 0.0px;text-transform: none;white-space: normal;word-spacing: 0.0px;">
<div>-- </div>
<div>
<div>Jayce Piel — <a href="mailto:jayce.piel@gmail.com" onclick="parent.window.location.href='jayce.piel@gmail.com'; return false;" target="_blank">jayce.piel@gmail.com</a> -- 0616762431</div>
<div> Responsable Informatique F.F.Tir</div>
</div>
</div>
</div>
</div>
</div>
-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: <a href="https://lists.samba.org/mailman/listinfo/rsync" target="_blank">https://lists.samba.org/mailman/listinfo/rsync</a> Before posting, read: <a href="http://www.catb.org/~esr/faqs/smart-questions.html" target="_blank">http://www.catb.org/~esr/faqs/smart-questions.html</a></div>
</div>
</div>
</div>
</div></div></body></html>