Odd behavior
Eberhard Moenkeberg
emoenke at gwdg.de
Thu Apr 22 16:38:42 MDT 2010
Hi,
On Thu, 22 Apr 2010, Erich Weiler wrote:
> Well, I solved this problem myself, it seems. It was not an rsync problem,
> per se, but it's interesting anyway on big filesystems like this so I'll
> outline what went down:
>
> Because my rsyncs were mostly just statting millions of files very quickly,
> RAM filled up with inode cache. At a certain point, the kernel stopped
> allowing new cache entries to be added to the slab memory it had been using,
> and was slow to reclaim memory on old, clean inode cache entries. So it
> basically slowed the I/O of the computer to barely anything.
>
> Slab memory can be checked by looking at the /proc/meminfo file. If you see
> that slab memory is using up a fair portion of your total memory, run the
> 'slabtop' program to see the top offenders. In my case, it was the
> filesystem that was screwing me (by way of the kernel).
>
> I was able to speed up the reclaiming of clean, unused inode cache entries by
> tweaking this in the kernel:
>
> # sysctl -w vm.vfs_cache_pressure=10000
>
> The default value for that is 100, where higher values release memory faster
> for dentries and inodes. It helped, but my rsyncs were still faster than it
> was, and it didn't help that much. What really fixed it was this:
>
> # echo 3 > /proc/sys/vm/drop_caches
>
> That effectively clears ALL dentry and inode entries from slab memory
> immediately. When I did that, memory usage dropped from 35GB to 500MB, my
> rsyncs fired themselves up again magically, and the computer was responsive
> again.
>
> Slab memory began to fill up again of course, as the rsyncs were still going.
> But slowly. For this edge case, I'm just going to configure a cron job to
> flush the dentry/inode cache every five minutes or so. But things look much
> better now!
>
> A word of warning for folks rsyncing HUGE numbers of files under linux. ;)
>
> As a side note, Solaris does not seem to have this problem, presumably
> because the kernel handles inode/dentry caching in a different way.
Thanks for your report. Really a mail to keep for future.
Viele Gruesse
Eberhard Moenkeberg (emoenke at gwdg.de, em at kki.org)
--
Eberhard Moenkeberg
Arbeitsgruppe IT-Infrastruktur
E-Mail: emoenke at gwdg.de Tel.: +49 (0)551 201-1551
-------------------------------------------------------------------------
Gesellschaft fuer wissenschaftliche Datenverarbeitung mbH Goettingen (GWDG)
Am Fassberg 11, 37077 Goettingen
URL: http://www.gwdg.de E-Mail: gwdg at gwdg.de
Tel.: +49 (0)551 201-1510 Fax: +49 (0)551 201-2150
Geschaeftsfuehrer: Prof. Dr. Bernhard Neumair
Aufsichtsratsvorsitzender: Dipl.-Kfm. Markus Hoppe
Sitz der Gesellschaft: Goettingen
Registergericht: Goettingen Handelsregister-Nr. B 598
-------------------------------------------------------------------------
More information about the rsync
mailing list