Samba4: LDB size limit and memory leak
Marcel.Ritter at rrze.uni-erlangen.de
Wed May 27 12:00:03 GMT 2009
Andrew Bartlett schrieb:
> On Mon, 2009-05-25 at 09:23 +0200, Marcel Ritter wrote:
>> Andrew Bartlett schrieb:
>>> On Mon, 2009-05-25 at 08:30 +0200, Marcel Ritter wrote:
>> Hi Andrew,
>>>> I'm running several samba 4 tests at the moment and I found two things,
>>>> that may need some attention:
>>>> 1. LDB size seems to be limited to 4GB (both on i586 and x86_64). I tried
>>>> to create a lot of new users using setup/newuser and somewhere in
>>>> 20000 and 40000 users, setup/newuser crashes:
>>> Correct. If a 64 bit varient of TDB were developed, that limit would be
>> So TDB is 32-bit only right now?
>> In bugzilla I found something about a 64-bit Patch to TDB, but it's quite
>> outdated (from 2005):
> Moving TDB to a 64 bit database is more about changing the on-disk
> format, and making improvements to the internal structure that would
> better suit a database of this size.
I'm trying to make tdb 64-bit capable - the tdb part looks promising so far
(no optimisations, just big files):
Creation of .tdb files > 4 GB works, tdbdump/tdbbackup works.
But: setup/newuser still stops working at 4GB file size,
not quite sure wether this is tdb, ldb or python related :-(
Unfortunately testing at this file size is very time consuming:
It takes several hours to make users.ldb grow > 4GB.
Any idea how to speed tests up is highly welcome.
> However, In retrospect, I'm surprised that you found the limit to be a
> 20,000 users. This seems rather small: We should expect users to only
> take up 2k each, with an additional overhead for indexes.
>> Any hint what needs to be done to make it 64-bit capable?
>>>> The resulting filesize of users.ldb is exactly 4GB + 4096 bytes
>>>> (4294971392 bytes)
> I think what's happening here is actually that you didn't add all the
> users in a single transaction, and as such are hitting up against the
> pathological database growth issues we first saw late last year.
Ok - size shrinks drastically to about 5% of the original size
(using /tmp/tdbbackup users.ldb) ->
-rw-r--r-- 1 root root 4421734400 26. Mai 21:14 users.ldb
-rw-r--r-- 1 root root 218132480 27. Mai 12:32 users.ldb.bak
> I think we need some internal trigger to say 'repack the database' (ie
> call tdb_repack())
> As a test, can you keep adding records if you call tdbbackup on the
> users.ldb (and then replace users.ldb with the backup)?
Adding users with setup/newuser works nicely again after running tdbbackup.
> Andrew Bartlett
Dipl.-Inf. Marcel Ritter
Unix _IS_ user friendly... It's just selective about who its friends are.
More information about the samba-technical