Samba4: LDB size limit and memory leak

[Andrew Bartlett]

On Mon, 2009-05-25 at 09:23 +0200, Marcel Ritter wrote:
> Andrew Bartlett schrieb:
> > On Mon, 2009-05-25 at 08:30 +0200, Marcel Ritter wrote:
> >   
> >> Hi,
> >>     
> Hi Andrew,
> >> I'm running several samba 4 tests at the moment and I found two things,
> >> that may need some attention:
> >>
> >> 1. LDB size seems to be limited to 4GB (both on i586 and x86_64). I tried
> >>     to create a lot of new users using setup/newuser and somewhere in
> >> between
> >>     20000 and 40000 users, setup/newuser crashes:
> >>     
> >
> > Correct.  If a 64 bit varient of TDB were developed, that limit would be
> > removed.
> >   
> So TDB is 32-bit only right now?
> 
> In bugzilla I found something about a 64-bit Patch to TDB, but it's quite
> outdated (from 2005):
> https://bugzilla.samba.org/show_bug.cgi?id=3270

Moving TDB to a 64 bit database is more about changing the on-disk
format, and making improvements to the internal structure that would
better suit a database of this size. 

However, In retrospect, I'm surprised that you found the limit to be a
20,000 users.  This seems rather small:  We should expect users to only
take up 2k each, with an additional overhead for indexes.  

> Any hint what needs to be done to make it 64-bit capable?
> >>    The resulting filesize of users.ldb is exactly 4GB + 4096 bytes
> >> (4294971392 bytes)
> >>

I think what's happening here is actually that you didn't add all the
users in a single transaction, and as such are hitting up against the
pathological database growth issues we first saw late last year.

I think we need some internal trigger to say 'repack the database' (ie
call tdb_repack())

As a test, can you keep adding records if you call tdbbackup on the
users.ldb (and then replace users.ldb with the backup)?

Andrew Bartlett

-- 
Andrew Bartlett                                http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org
Samba Developer, Red Hat Inc.                  http://redhat.com

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://lists.samba.org/archive/samba-technical/attachments/20090527/aff7b043/attachment.bin