lmdb file size fixed at 8GiB

[Tim Beale]

Hi Denis,

I was just wondering when do you see this problem? Is it when you're
populating the database, or is it when you're joining a new DC? And are
you testing on 4.9 or 4.10rc/master?

We noticed a similar problem with LMDB when testing a domain with a
large numbers of users. For us, it always happened during the DRS
replication of a join. It seemed to be related to how LMDB buffers a
large number of modifications to the same object during a single DB
transaction. This happens during DRS when applying the linked attributes
(i.e. group memberships), when joining a DC for the first time.

We tried increasing the 8Gb limit to avoid the problem, but (from
memory) we still hit the problem when we added more users. In the end,
we could optimize DRS to avoid modifying the same object multiple times.
This was better for efficiency and avoided the LMDB problem. This
particular problem should be fixed in 4.10, i.e.
https://git.samba.org/?p=samba.git;a=commit;h=775054afbe15129691c7

Cheers,
Tim

On 1/02/19 12:58 AM, Denis Cardon via samba-technical wrote:
> Hi everyone,
>
> I have a client preparing its switch to LMDB backend and in the
> testing environement we have reached the 8GiB hard limit in ldb_mdb.c
> [1].
>
> It is not very hard to rebuild the RPMS with a 16GiB limit, however I
> had two question in mind :
>
> * would it be possible to provide the size during the DC
> provisioning/join (it would be fixed on a a single DC, but it could be
> increase by joining a new one and removing the old one). And it would
> perhaps help for the bugzilla 13630 [2]
>
> * if I change the mmap_size, is there anything I should be careful
> about if the lmdb file has been initially created with a 8GiB limit?
>
> Cheers,
>
> Denis
>
> [1]
> https://gitlab.com/samba-team/samba/blob/samba-4.9.4/lib/ldb/ldb_mdb/ldb_mdb.c#L681
> [2] We could set a smaller size on 32bit system
> https://bugzilla.samba.org/show_bug.cgi?id=13630
>
>