[LDB PATCH]optimize ldb performance

[simo]

This Week End I have reworked once again a patch about making ldn_dn
manipulation more efficient by keeping the DN as strings as long as
possible and actually not exploding the DN until it is necessary to
perform some operation.

During the work I found out we needed better manipulation functions and
also some of the existing one were misused generating very inefficient
and memory hungry code.

Many functions in ldb_dn has been replace and rewritten from scratch
(almost all), some of them have been removed and new ones have been
added.

I think this patch not only makes raw ldb search up to 30% faster but
also make up a much saner interface to deal with DNs.

I'm planning to commit this shortly if there is no opposition, but I'd
like to see comments on the API changes I made.

Note for example that now struct ldb_dn embeds a pointer to ldb_context,
this may seem a bit strange but as we need an ldb_contect to casefold a
DN and as casefolding may now be implicit I preferred to refernce the
ldb context directly into the structire then requiring to pass and ldb
context on every possible operation performed on an struct ldb_dn. This
will probably also make it possibile to remove the custome qsort
function we have in ldb as passing the ldb_context I think was the only
reason to make it different from the other ones available.

Note also that the code is not 100% finalized, while it compiles and
passes all tests it also introduced one or more memleaks somewhere, I am
going to work on fix these and other minor things after the commit.


Attached you can also see the numbers of the LOCAL-DBSPEED test before
making struct ldb_dn an opaque pointer, after that and after applying
this optimization patch.

Simo.
-- 
Simo Sorce
Samba Team GPL Compliance Officer
email: idra at samba.org
http://samba.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: samba4_optimize2_struct_ldb_dn.patch
Type: text/x-patch
Size: 182633 bytes
Desc: not available
Url : http://lists.samba.org/archive/samba-technical/attachments/20061121/81cc46df/samba4_optimize2_struct_ldb_dn-0001.bin
-------------- next part --------------
Non-opaque struct ldb_dn

$ ./bin/smbtorture $CONFIGURATION $TORTURE_OPTIONS $AUTH //$SERVER/test LOCAL-DBSPEED
Using seed 1162497352
Running DBSPEED
Testing tdb speed for sidmap
Adding 1000 SID records
Testing for 10 seconds
tdb speed 91322.15 ops/sec
Testing ldb speed for sidmap
Adding 1000 SID records
Testing for 10 seconds
ldb speed 8334.91 ops/sec
DBSPEED took 20.5531 secs

Tests: 2, Failures: 0, Errors: 0, Skipped: 0. Success rate: 100.00%


OPAQUE struct ldb_dn

$ ./bin/smbtorture $CONFIGURATION $TORTURE_OPTIONS $AUTH //$SERVER/test LOCAL-DBSPEED
Using seed 1162424057
Running DBSPEED
Testing tdb speed for sidmap
Adding 1000 SID records
Testing for 10 seconds
tdb speed 93857.36 ops/sec
Testing ldb speed for sidmap
Adding 1000 SID records
Testing for 10 seconds
ldb speed 8181.59 ops/sec
DBSPEED took 20.4993 secs

Tests: 2, Failures: 0, Errors: 0, Skipped: 0. Success rate: 100.00%

Optimize 2

$ ./bin/smbtorture $CONFIGURATION $TORTURE_OPTIONS $AUTH //$SERVER/test LOCAL-DBSPEED
Using seed 1164086843
Running DBSPEED
Testing tdb speed for sidmap
Adding 1000 SID records
Testing for 10 seconds
tdb speed 91456.44 ops/sec
Testing ldb speed for sidmap
Adding 1000 SID records
Testing for 10 seconds
ldb speed 10899.56 ops/sec
DBSPEED took 20.4855 secs