Samba4: LDB size limit and memory leak

[Marcel Ritter]

Andrew Bartlett schrieb:
> On Mon, 2009-05-25 at 09:23 +0200, Marcel Ritter wrote:
>   
>> Andrew Bartlett schrieb:
>>     
>>> On Mon, 2009-05-25 at 08:30 +0200, Marcel Ritter wrote:
>>>   
>>>       
>>>> Hi,
>>>>     
>>>>         
>> Hi Andrew,
>>     
>>>> I'm running several samba 4 tests at the moment and I found two things,
>>>> that may need some attention:
>>>>
>>>> 1. LDB size seems to be limited to 4GB (both on i586 and x86_64). I tried
>>>>     to create a lot of new users using setup/newuser and somewhere in
>>>> between
>>>>     20000 and 40000 users, setup/newuser crashes:
>>>>     
>>>>         
>>> Correct.  If a 64 bit varient of TDB were developed, that limit would be
>>> removed.
>>>   
>>>       
>> So TDB is 32-bit only right now?
>>
>> In bugzilla I found something about a 64-bit Patch to TDB, but it's quite
>> outdated (from 2005):
>> https://bugzilla.samba.org/show_bug.cgi?id=3270
>>     
>
> Moving TDB to a 64 bit database is more about changing the on-disk
> format, and making improvements to the internal structure that would
> better suit a database of this size.
>   
I'm trying to make tdb 64-bit capable - the tdb part looks promising so far
(no optimisations, just big files):
Creation of .tdb files > 4 GB works, tdbdump/tdbbackup works.

But: setup/newuser still stops working at 4GB file size,
not quite sure wether this is tdb, ldb or python related :-(

Unfortunately testing at this file size is very time consuming:
It takes several hours to make users.ldb grow > 4GB.

Any idea how to speed tests up is highly welcome.
> However, In retrospect, I'm surprised that you found the limit to be a
> 20,000 users.  This seems rather small:  We should expect users to only
> take up 2k each, with an additional overhead for indexes.  
>
>   
>> Any hint what needs to be done to make it 64-bit capable?
>>     
>>>>    The resulting filesize of users.ldb is exactly 4GB + 4096 bytes
>>>> (4294971392 bytes)
>>>>
>>>>         
>
> I think what's happening here is actually that you didn't add all the
> users in a single transaction, and as such are hitting up against the
> pathological database growth issues we first saw late last year.
>   
Ok - size shrinks drastically to about 5% of the original size
(using /tmp/tdbbackup  users.ldb) ->
-rw-r--r-- 1 root root 4421734400 26. Mai 21:14 users.ldb
-rw-r--r-- 1 root root  218132480 27. Mai 12:32 users.ldb.bak

> I think we need some internal trigger to say 'repack the database' (ie
> call tdb_repack())
>
> As a test, can you keep adding records if you call tdbbackup on the
> users.ldb (and then replace users.ldb with the backup)?
>   
Adding users with setup/newuser works nicely again after running tdbbackup.
> Andrew Bartlett
>   
Bye,
   Marcel

-- 
Dipl.-Inf. Marcel Ritter
----
Unix _IS_ user friendly... It's just selective about who its friends are.