[TDB] Patches for file and memory usage growth issues
idra at samba.org
Tue Apr 12 07:01:25 MDT 2011
On Mon, 2011-04-11 at 20:50 +1000, tridge at samba.org wrote:
> Hi Simo,
> > during this week I have been working on a memory growth issue affecting
> > sssd when storing large group of users in the LDB based cache.
> Can you give us an idea of the number of records involved, the number
> of indexed attributes and how big the records are?
See the info in the mail with numbers I posted in response to Rusty.
Briefly, 25k users, and 250 groups, a total of ~75k records varying in
size from a few KiB to a few MiB.
> > The first one should be uncontroversial, the original intent makes sense
> > in a tdb database when you use small records, but when a rare huge
> > record is saved in, growing the db by 100 times its size really is
> > overkill. I also reduced the minimum overhead from 25% to 10% as with
> > large TDBs (in the order of 500MB/1GB, growing by 25% is quite a lot).
> The change from 25% to 10% will have the bigger impact of the two
> changes I think. The 25% number was always fairly arbitrary, but
> changing to 10% means 2.5 times as many tdb_expand calls while a
> database grows. Have you measured any impact of this in initially
> creating a large database?
I have implemented the heuristic Rusty proposed so that 25% more is
allocated for small dbs while for dbs above 100 megabytes we switch to
allocate only 10% more.
> > The second one needs discussion. I noticed that a transaction can
> > substantially increase the size of a tdb, in my tests I had a 1.2G size
> > TDB actually grow to a ~1.7G after a backup ... by not using a
> > transaction to store the backup copy the same 1.2G TDB shrunk down to
> > ~900MB. I know that the reason we use a transaction is to avoid rsyncs
> > of partial DBs, so maybe we want to add an option on whether to use a
> > transaction for the 2nd db ? Comments welcome.
> I initially thought you were removing the transaction on the old db,
> but I see that you're only removing it on the new db. That should be
> OK, but we will need to at least get a write lock across the whole of
> tdb_new, and we should probably also do a fsync at the end on tdb_fd()
> before the close and rename.
I added a tdb_locakall() and a final f[data]sync() on tdb_fd().
Please let me know if these patches are ok as is and I'll push them.
Samba Team GPL Compliance Officer <simo at samba.org>
Principal Software Engineer at Red Hat, Inc. <simo at redhat.com>
More information about the samba-technical