tdb memory consumption

tridge at samba.org tridge at samba.org
Mon Feb 12 21:42:55 GMT 2007


Volker,

 > The following patch adds a dynamic allocation of the
 > tdb->locked[] array. I'm not sure if we can max it at a
 > certain number, so that we could change it to a static
 > array.

I think that for a long lived smbd that your patch will increase
memory usage quite a lot, and slow it down a lot, which isn't what we
want :-)

Let's think about (say) opendb.tdb. For a long lived smbd it will
eventually hit every tdb hash chain at least once, assuming
device/inode numbers are evenly distributed. If we assume a hash size
of 10k (same as your example), that means the memory usage will grow
to:

  (10000 * 12) + (10000 * 4) + (10000 * 16) = 320k per tdb

this assumes malloc has a 16 byte overhead per allocation (should be
about right), and pointers are 4 bytes. So that's 640k for the 2 tdb's
in your example.

You'll also end walking along a 10k long list finding the right lock
record for the right list, so the lock call will get much slower.

I suspect you meant to re-use elements from the lockrecs when their
count is zero. You could do that like this:

change from:
		if (tdb->lockrecs[i].list == list) {
			if (tdb->lockrecs[i].count == 0) {

to:
		if (tdb->lockrecs[i].list == list ||
		   tdb->lockrecs[i].count == 0) {
			tdb->lockrecs[i].list = list;

that would at least prevent the looping along that array for empty
elements on every lock (or is that avoided somewhere else in your
patch that I missed? my apologies if so)

Another alternative would be to use a bitmap with 2 bits per lock, and
only allocate elements in an overflow array when the lock count
reaches 3. As recursive locks are rare, we would in all likelyhood use
just 2 bits per hash chain, so 2.5k per open tdb. That would also
avoid the loop and be very fast even for large numbers of records
being locked.

So each 2 bit pair would take on values like this:

  00 : not locked
  01 : 1 lock
  10 : 2 locks
  11 : more than 2 locks, look in the overflow array

we'd also need a 1 bit per entry array for the ltype. So total memory
usage in the unlocked case would be 3 bits per hash chain. In the
worst case (all hash chains have 3 or more locks) it will still use
less memory than your patch, as we use only 1 bit for the ltype.

Cheers, Tridge


More information about the samba-technical mailing list