ldbadd --no-sync implications?

[Michael Wood]

Hi again

On 3 March 2011 18:38, Michael Wood <esiotrot at gmail.com> wrote:
> Hi metze
>
> On 3 March 2011 18:32, Stefan (metze) Metzmacher <metze at samba.org> wrote:
>> Hi Michael,
>>
>>> By the way, adding batches of 1000 users to the group takes longer and long.
>>>
>>> e.g. if I have LDIF files like this:
>>>
>>> dn: CN=TestGroup,CN=Users,DC=example,DC=com
>>> changetype: modify
>>> -
>>> add: member
>>> member: CN=one,CN=Users,DC=example,DC=com
>>> -
>>> add: member
>>> member: CN=two,CN=Users,DC=example,DC=com
>>> -
>>> add: member
>>> member: CN=three,CN=Users,DC=example,DC=com
>>> -
>>> add: member
>>> member: CN=four,CN=Users,DC=example,DC=com
>>> -
>>
>> My ldif looks like this:
>>
>> dn: CN=TestGroup,CN=Users,DC=example,DC=com
>> changetype: modify
>> add: member
>> member: CN=one,CN=Users,DC=example,DC=com
>> member: CN=two,CN=Users,DC=example,DC=com
>> member: CN=three,CN=Users,DC=example,DC=com
>> member: CN=four,CN=Users,DC=example,DC=com
>> -
>>
>> But I guess it doesn't make a difference.
>
> I'll give it a try and see if it helps.
>
>>> with 1000 users in each, and then run that through ldbmodify --nosync,
>>> I get the following times.  As you can see, each batch of 1000 users
>>> gets slower and slower:
>>
>> I think I have fixed that with commits:
>>
>> 05088fb855a1fc043c3f75d01742cdbbfbb3330e s4:dsdb/samldb: avoid nested
>> unindexed searches in samldb_member_check()
>> 60691c13224656e6a889eb01246582f10c056b7b s4:dsdb/samdb: optimize
>> samldb_prim_group_change()
>
> Unfortunately the slowing down of ldbmodify for adding more and more
> batches of 1000 users to a group with a large number of users in it
> still seems to be a problem.
>
> I'm now running Version 4.0.0alpha15-GIT-ba77274 on a test box, and
> since the upgrade it's still getting slower:
>
> batch-csv.bd-group.ldif
> Modified 1 records with 0 failures
>
> real    9m12.581s
> user    8m58.730s
> sys     0m8.960s
> batch-csv.be-group.ldif
> Modified 1 records with 0 failures
>
> real    9m19.655s
> user    9m11.270s
> sys     0m7.020s
> batch-csv.bf-group.ldif
> Modified 1 records with 0 failures
>
> real    9m45.768s
> user    9m37.520s
> sys     0m6.900s
>
> I haven't tried yet to see whether OS X is any happier with the new
> version, though.

I've just tried it with a similar ldif to yours, and it makes a *huge*
difference:

Modified 1 records with 0 failures

real	0m28.793s
user	0m20.740s
sys	0m7.960s

Thanks!

I'll test adding batches of users again and see if that's changed with
the new version.  The above was all to do with adding the users to a
group.

I'll also see what OS X thinks of the new version and try the
LDB_WARN_UNINDEXED environment variable to see if it tells me
anything.

-- 
Michael Wood <esiotrot at gmail.com>