[TDB] Patches for file and memory usage growth issues

simo idra at samba.org
Tue Apr 12 07:01:25 MDT 2011


On Mon, 2011-04-11 at 20:50 +1000, tridge at samba.org wrote:
> Hi Simo,
> 
>  > during this week I have been working on a memory growth issue affecting
>  > sssd when storing large group of users in the LDB based cache.
> 
> Can you give us an idea of the number of records involved, the number
> of indexed attributes and how big the records are?

See the info in the mail with numbers I posted in response to Rusty.
Briefly, 25k users, and 250 groups, a total of ~75k records varying in
size from a few KiB to a few MiB.

>  > The first one should be uncontroversial, the original intent makes sense
>  > in a tdb database when you use small records, but when a rare huge
>  > record is saved in, growing the db by 100 times its size really is
>  > overkill. I also reduced the minimum overhead from 25% to 10% as with
>  > large TDBs (in the order of 500MB/1GB, growing by 25% is quite a lot).
> 
> The change from 25% to 10% will have the bigger impact of the two
> changes I think. The 25% number was always fairly arbitrary, but
> changing to 10% means 2.5 times as many tdb_expand calls while a
> database grows. Have you measured any impact of this in initially
> creating a large database?

I have implemented the heuristic Rusty proposed so that 25% more is
allocated for small dbs while for dbs above 100 megabytes we switch to
allocate only 10% more.

See:
http://git.samba.org/?p=idra/samba.git;a=commitdiff;h=a2c8b1e240c788bc145544c9630a7d4a21ff5241

>  > The second one needs discussion. I noticed that a transaction can
>  > substantially increase the size of a tdb, in my tests I had a 1.2G size
>  > TDB actually grow to a ~1.7G after a backup ... by not using a
>  > transaction to store the backup copy the same 1.2G TDB shrunk down to
>  > ~900MB. I know that the reason we use a transaction is to avoid rsyncs
>  > of partial DBs, so maybe we want to add an option on whether to use a
>  > transaction for the 2nd db ? Comments welcome.
> 
> I initially thought you were removing the transaction on the old db,
> but I see that you're only removing it on the new db. That should be
> OK, but we will need to at least get a write lock across the whole of
> tdb_new, and we should probably also do a fsync at the end on tdb_fd()
> before the close and rename.

I added a tdb_locakall() and a final f[data]sync() on tdb_fd().

See:
http://git.samba.org/?p=idra/samba.git;a=commitdiff;h=54ac2dd19b9187089d2ee86a226f65f634bf660a


Please let me know if these patches are ok as is and I'll push them.

Simo.

-- 
Simo Sorce
Samba Team GPL Compliance Officer <simo at samba.org>
Principal Software Engineer at Red Hat, Inc. <simo at redhat.com>



More information about the samba-technical mailing list