sync performance falls off a cliff
leen at consolejunky.net
Sun Jul 5 08:17:14 GMT 2009
> >I'm no expert, but I suggest using rsync 3.x (3.0.6 for example), it
> >doesn't keep the as much information of the filelist in memory.
> Yes. Or at lease it starts transfers much faster, because it doesn't
> wait for the full list to be completed.
> >It's probably swapping to disk, because of the large list and that
> >significantly slows down the performance of the whole machine(s).
> He's probably running out of ram, not only because of rsync but
> also everything else. Since inodes and files are not in ram, they have
> to be fetched from the disk, which is *very* slow.
not ram per se, because this is what he said in a different e-mail:
Today I've been watching the production 2.6.8 rsync off and on and no it
isn't swapping. Used "vmstat" and "top" both on the source and
the destination. Each shows 0 for si and so.
> You can tell the kernel to increase the priority of inodes, which will
> reduce the time to build the file list a lot. Just set
> /proc/sys/vm/vfs_cache_pressure to a low value.
Ohh, interresting, thank you.
I did already suggest ionice.
I guess we'll have to see how it goes, because he hasn't put it in
More information about the rsync