--stats and performance issue

Sun Jun 10 01:19:20 MDT 2012

Hello,

My question is regarding the interpretation of --stats output and 
whether I have a performance issue or not.

Can the client/server architecture (rsync client on one machine, rsync 
server on another machine) be far slowlier than a local rsync session ?

This is the ouput of an rsync session:

date
rsync -v --recursive --links --perms --times --group --owner --xattrs 
--acls --delete --stats --human-readable \
     --link-dest=/storage/daily/home/d1 \
     serveur3::home /storage/daily/home/d0
date

sam. juin  9 07:10:19 CEST 2012

receiving incremental file list
./
aquota.group
aquota.user
accueil/COURRIER/RETOUR DOC. ORIGINAUX .doc
accueil/COURRIER/demande liste papiers rue du marché.doc
accueil/COURRIER/demande par courrier MME.doc
accueil/COURRIER/plusieurs dem. par tél..doc
accueil/Document Logement/Répertoire GT.xls
accueil/Logements Vacants/Logements vacants 2010/LOGEMENTS VACANTS 2010 
JUIL à DEC.xls
accueil/Logements Vacants/Logements vacants 2010/Logements vacants CA 
2010.xls
accueil/ilot gambetta/ACCORLO SAINT AUBIN LA POSTE SANS GT.doc
commercial/contentieux/PORTE VUE LOGEMENTS/PARC SAINT CYR/013 Parc Saint 
Cyr - Les Cyclamens.xls
commercial/contentieux/PORTE VUE LOGEMENTS/PARC SAINT CYR/013 Parc Saint 
Cyr - Les Géraniums.xls
commercial/contentieux/PORTE VUE LOGEMENTS/PARC SAINT CYR/013 Parc Saint 
Cyr - Oasis.xls
kavupdater/
kavupdater/retranslator/retranslator.log
profiles/clambany/Application 
Data/Microsoft/CryptnetUrlCache/Content/7B2238AACCEDC3F1FFE8E7EB5F575EC9
profiles/clambany/Application 
Data/Microsoft/CryptnetUrlCache/MetaData/7B2238AACCEDC3F1FFE8E7EB5F575EC9
sahlm/Formation EXCEL.xls
technique/5D666000
technique/EDL 2011.xls
technique/SINISTRES 2010   2011/T DOSSIERS SINISTRES DE 02 à 2011/TABL 
SINI 2010  goupil.xls
technique/a.Clément/SUIVI DES DEVIS 2010.xls
technique/a.Clément/SUIVI DES DEVIS 2011.xls
technique/a.Clément/SUIVI DES DEVIS 2012.xls
technique/a.Teddy/EDL/EDL 2008.xls

Number of files: 1117407
Number of files transferred: 5
Total file size: 202.86G bytes
Total transferred file size: 20.67K bytes
Literal data: 1.67K bytes
Matched data: 19.00K bytes
File list size: 30.32M
File list generation time: 0.099 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 114.17K
Total bytes received: 30.69M

sent 114.17K bytes  received 30.69M bytes  43.23K bytes/sec
total size is 202.86G  speedup is 6586.34

sam. juin  9 07:22:11 CEST 2012

THIS IS ROUGHLY 1600 FILES / SECOND PROCESSED
*********************************************

The server is a fast machine with SCSI disks, connected through a 1Gbs 
link to the client.
The client is a new dedicated backup machine with standard dedicated 
SATA disks

# hdparm -t /dev/sdc

/dev/sdc:
  Timing buffered disk reads:  446 MB in  3.01 seconds = 148.31 MB/sec

CPU stats during a rsync session:
on the client (top), rsync two processes are usually on top
load average 0.11 0.31 0.36
rsync process consumes between 1-50% CPU

I did some other test without --dest-link option

date
rsync --recursive --links --perms --times --group --owner --xattrs 
--acls --delete --stats --human-readable \
     serveur3::home /storage/monthly/home
date

Run 1 (destination directory was empty):
*****************
sam. juin  9 07:34:08 CEST 2012

Number of files: 1117487
Number of files transferred: 1003718
Total file size: 207.09G bytes
Total transferred file size: 207.09G bytes
Literal data: 207.09G bytes
Matched data: 0 bytes
File list size: 30.32M
File list generation time: 0.695 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 19.76M
Total bytes received: 207.19G

sent 19.76M bytes  received 207.19G bytes  29.34M bytes/sec
total size is 207.09G  speedup is 1.00

sam. juin  9 09:31:50 CEST 2012

Load average was 0.17 0.19 0.26

Run 2:
*****************
sam. juin  9 10:14:25 CEST 2012

Number of files: 1117487
Number of files transferred: 21
Total file size: 207.09G bytes
Total transferred file size: 4.14G bytes
Literal data: 124.38M bytes
Matched data: 4.01G bytes
File list size: 30.32M
File list generation time: 0.054 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 1.51M
Total bytes received: 155.86M

sent 1.51M bytes  received 155.86M bytes  390.99K bytes/sec
total size is 207.09G  speedup is 1315.90

sam. juin  9 10:21:07 CEST 2012

THIS IS ROUGHLY 2700 FILES / SECOND PROCESSED
*********************************************

I did a third test on my home machine which is using rsync for backup on 
a *locally attached USB disk* (sdc):

rsync options: '-a --delete --delete-excluded --one-file-system --stats 
--human-readable'

# hdparm -t /dev/sdc1

/dev/sdc1:
  Timing buffered disk reads:  70 MB in  3.02 seconds =  23.17 MB/sec

dim. juin 10 08:52:55 CEST 2012

Number of files: 107883
Number of files transferred: 0
Total file size: 564.75G bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 2.40M
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 2.40M
Total bytes received: 2.27K

sent 2.40M bytes  received 2.27K bytes  177.86K bytes/sec
total size is 564.75G  speedup is 235206.08

dim. juin 10 08:53:08 CEST 2012

THIS IS ROUGHLY 107000 FILES / SECOND PROCESSED
***********************************************

40 TIMES FASTER !!!!

Thanks a lot for your kind advise