sync performance falls off a cliff

[Carlos Carvalho]

Leen Besselink (leen at consolejunky.net) wrote on 5 July 2009 10:17:
 >>  >
 >>  >I'm no expert, but I suggest using rsync 3.x (3.0.6 for example), it
 >>  >doesn't keep the as much information of the filelist in memory.
 >> 
 >> Yes. Or at lease it starts transfers much faster, because it doesn't
 >> wait for the full list to be completed.
 >> 
 >>  >It's probably swapping to disk, because of the large list and that
 >>  >significantly slows down the performance of the whole machine(s).
 >> 
 >> He's probably running out of ram, not only because of rsync but
 >> also everything else. Since inodes and files are not in ram, they have
 >> to be fetched from the disk, which is *very* slow.
 >> 
 >
 >not ram per se, because this is what he said in a different e-mail:
 >
 >Today I've been watching the production 2.6.8 rsync off and on and no it
 >isn't swapping. Used "vmstat" and "top" both on the source and
 >the destination. Each shows 0 for si and so.

He *is* running out of ram for cache, so the machine has to get the
pages from disk, which is slow. Swap is not used because the pages are
not modified.