Odd behavior

Eberhard Moenkeberg emoenke at gwdg.de
Thu Apr 22 16:38:42 MDT 2010


Hi,

On Thu, 22 Apr 2010, Erich Weiler wrote:

> Well, I solved this problem myself, it seems.  It was not an rsync problem, 
> per se, but it's interesting anyway on big filesystems like this so I'll 
> outline what went down:
>
> Because my rsyncs were mostly just statting millions of files very quickly, 
> RAM filled up with inode cache.  At a certain point, the kernel stopped 
> allowing new cache entries to be added to the slab memory it had been using, 
> and was slow to reclaim memory on old, clean inode cache entries.  So it 
> basically slowed the I/O of the computer to barely anything.
>
> Slab memory can be checked by looking at the /proc/meminfo file.  If you see 
> that slab memory is using up a fair portion of your total memory, run the 
> 'slabtop' program to see the top offenders.  In my case, it was the 
> filesystem that was screwing me (by way of the kernel).
>
> I was able to speed up the reclaiming of clean, unused inode cache entries by 
> tweaking this in the kernel:
>
> # sysctl -w vm.vfs_cache_pressure=10000
>
> The default value for that is 100, where higher values release memory faster 
> for dentries and inodes.  It helped, but my rsyncs were still faster than it 
> was, and it didn't help that much.  What really fixed it was this:
>
> # echo 3 > /proc/sys/vm/drop_caches
>
> That effectively clears ALL dentry and inode entries from slab memory 
> immediately.  When I did that, memory usage dropped from 35GB to 500MB, my 
> rsyncs fired themselves up again magically, and the computer was responsive 
> again.
>
> Slab memory began to fill up again of course, as the rsyncs were still going. 
> But slowly.  For this edge case, I'm just going to configure a cron job to 
> flush the dentry/inode cache every five minutes or so.  But things look much 
> better now!
>
> A word of warning for folks rsyncing HUGE numbers of files under linux.  ;)
>
> As a side note, Solaris does not seem to have this problem, presumably 
> because the kernel handles inode/dentry caching in a different way.

Thanks for your report. Really a mail to keep for future.


Viele Gruesse
Eberhard Moenkeberg (emoenke at gwdg.de, em at kki.org)

-- 
Eberhard Moenkeberg
Arbeitsgruppe IT-Infrastruktur
E-Mail: emoenke at gwdg.de      Tel.: +49 (0)551 201-1551
-------------------------------------------------------------------------
Gesellschaft fuer wissenschaftliche Datenverarbeitung mbH Goettingen (GWDG)
Am Fassberg 11, 37077 Goettingen
URL:    http://www.gwdg.de             E-Mail: gwdg at gwdg.de
Tel.:   +49 (0)551 201-1510            Fax:    +49 (0)551 201-2150
Geschaeftsfuehrer:           Prof. Dr. Bernhard Neumair
Aufsichtsratsvorsitzender: Dipl.-Kfm. Markus Hoppe
Sitz der Gesellschaft:     Goettingen
Registergericht:           Goettingen  Handelsregister-Nr. B 598
-------------------------------------------------------------------------


More information about the rsync mailing list