[PATCHES v2 BUG 13865] Properly track the size of talloc objects

Christof Schmitt cs at samba.org
Fri Apr 5 22:40:56 UTC 2019


On Fri, Apr 05, 2019 at 03:39:50PM -0700, Jeremy Allison wrote:
> On Fri, Apr 05, 2019 at 03:34:17PM -0700, Christof Schmitt wrote:
> > On Fri, Apr 05, 2019 at 03:15:17PM -0700, Jeremy Allison wrote:
> > > 
> > > One more point. In case someone thinks
> > > setting "max stat cache size = 8192"
> > > will cause us to use more memory, just
> > > remember right now without Christof's
> > > patch we're not accounting for *any*
> > > of the talloc memory being stored in
> > > the cache, beyond an 8 byte pointer
> > > per object. Which means currently
> > > the memcache is essentially unlimited
> > > in the memory it can cache :-).
> > > 
> > > I just want us to set a new reasonable
> > > default :-).
> > 
> > Anything we choose is a guess, since every workload is different. On a
> > quick test to open multiple files with smbclient and then checking size
> > allocated under 'struct memcache' in smbcontrol pool-usage, shows that
> > each open file increases the memory usage by something like 300 to 700
> > bytes.
> > 
> > So that 16k for each share mode seems high, although that might increase
> > with concurrent access to the same file. 4k for each getwd cache also
> > seems high, that seems to mainly depend on the length of the path name.
> > 
> > Rounding the share mode size up to 1024 bytes and the getwd length to
> > 1024, and assuming one getwd cache for each open file (probably also too
> > high), that would yield 100 * 1024 + 100 * 1024 = 200k
> 
> The getwd cache is a singleton (it's the current working directory),
> it's not per open file.
> 
> > That would be more in line with doubling the state cache size from 256
> > to 512 (kB).
> > 
> > In the end i am fine with any number, just trying to get a feeling for
> > the actual space used here.
> 
> Yes, the workload mix also matters. The 'stat cache' canonicalization
> for pathnames also uses the global memcache (just not via talloc),
> so if you have a workload that's heavy on pathname processing
> then the cache will (already) fill up with pathname canonicalization.
> 
> Right now the talloc entries essentially aren't counted, so we
> get everything in them for free.
> 
> So we do need a bigger default I think to avoid elements being
> evicted too often. How about making it 1MB instead of 8Mb
> (max stat cache size = 1024) ?

Seems reasonable. I can resend the series with an additional patch to
increase the default to 1024.

Christof



More information about the samba-technical mailing list