[TDB] Patches for file and memory usage growth issues

simo idra at samba.org
Sat Apr 9 21:01:15 MDT 2011


Hi all,
during this week I have been working on a memory growth issue affecting
sssd when storing large group of users in the LDB based cache.

It turns out that a lot of memory is being used by TDB.

Please take a look at the last 3 patches in this tree:
http://git.samba.org/?p=idra/samba.git;a=shortlog;h=refs/heads/tdb-memsize

The first one should be uncontroversial, the original intent makes sense
in a tdb database when you use small records, but when a rare huge
record is saved in, growing the db by 100 times its size really is
overkill. I also reduced the minimum overhead from 25% to 10% as with
large TDBs (in the order of 500MB/1GB, growing by 25% is quite a lot).

The second one needs discussion. I noticed that a transaction can
substantially increase the size of a tdb, in my tests I had a 1.2G size
TDB actually grow to a ~1.7G after a backup ... by not using a
transaction to store the backup copy the same 1.2G TDB shrunk down to
~900MB. I know that the reason we use a transaction is to avoid rsyncs
of partial DBs, so maybe we want to add an option on whether to use a
transaction for the 2nd db ? Comments welcome.

The third one I think is less controversial but still not something I
would put in without discussion. This feature has the very nice property
of being completely optional and needs an explicit flags to be passed at
tdb_open(). This makes it relatively harmless and not very intrusive.
Basically this patch uses zlib with raw deflate/inflate to compress any
record data that is higher then a specific threshold (currently 16kb).
For example an ldb index gets easily compressed down to less than 10%
the original size, for obvious reasons, although compressing an index is
not necessarily a too bright idea, yet ...
Using this option with some tests we are doing in sssd with big groups
and indexes (25k users as members for 250 groups, and indexed on
objectclass=user), I've got the tdb size shrink 30%, although at the
cost of a performance hit of a 30% (not doing searches in this test only
storing users and groups with huge memberships). The runtime memory used
also dropped around 40% which was what I was actually after: we are
hitting a case where LDB is using 3.5G of RAM on a dataset of 650MB
uncompressed (the actual TDB size being currently 30% bigger at ~900MB
with a lot of free space due to tdb_expand() behavior which in my tests
still did 25% expansions minimum (now changed to 10% in the first
patch). With compression the dataset shrinks to ~480M with file still
being 30% bigger for the same reason.

I am still not sure I will actually use compression rather then changing
how we store indexes and memberships in the long term, but because the
option is not invasive at all and people can choose when to use it, it
seem like a potentially nice feature for those cases where speed is less
important than file size.

Comments are very welcome, if there aren't strong objections I'd like to
push patch 1 and 3 (tdb_expand fixes and tdb compression) during the
week. The tdbbackup one is probably controversial and will probably need
more work, so I will not push it unless explicitly acked as is.


Finally I also am going to try to come to a solution for the huge memory
usage caused by tdb_repack(). This function actually needs a lot of
improvement. Firstly because it uses a transaction to copy the database
twice do the repack.
This is inefficient in 3 ways.
1. Temporarily doubles the memory used (due to the second tdb in
memory), which is a huge problem when your TDB file is *big* (ldb use
cases with many users/groups).
2. It actually does 2 copies of the TDB back and forth after wiping all
the records of the original tdb, all whithin a transaction (lots of
memory used).
3. Due to the transaction it can sometimes increase the actual size of
the tdb file instead of shrinking it. I have some ideas of why it is
happening but haven't analyzed it fully yet (seem like the same issue
affecting tdbbackup except we cannot avoid using a transaction here as
quick dirty fix like I did in patch 2).
So if anyone have ideas on how to deal with this they are really welcome
as the repack is killing our machines (OOM Killer getting in action in
some cases, plus slowness), although disabling tdb_repack is also not an
option as the tdb becomes quickly fragmented with many big transactions
containing huge records and grows even more.

cheers,
Simo.

-- 
Simo Sorce
Samba Team GPL Compliance Officer <simo at samba.org>
Principal Software Engineer at Red Hat, Inc. <simo at redhat.com>



More information about the samba-technical mailing list