[PATCH] Core ldb modules changes

[simo]

On Fri, 2008-01-18 at 10:41 +1100, Andrew Bartlett wrote:
> On Thu, 2008-01-17 at 18:26 -0500, simo wrote:
> > On Fri, 2008-01-18 at 10:25 +1100, Andrew Bartlett wrote:
> > > On Thu, 2008-01-17 at 10:37 -0500, simo wrote:
> > > > This is a new patch[1] to the core ldb internal async API to address some
> > > > issue that were preventing me from using ldb in a full async server.
> > > > The next step will be to add support to replace the run queue now
> > > > attached to the ldb context with an event system and completely stop
> > > > using the ldb_wait() call in fully async code calling code.
> > > 
> > > > Before committing I will finish porting the dsdb modules to this
> > > > interface, but I thought that people may be interested in commenting and
> > > > understanding the direction I want to take with this code.
> > > > 
> > > > Simo.
> > > > 
> > > > [1] Patch here as it is a bit too big for the list:
> > > > http://samba.org/~idra/patches/0001-This-commit-changes-in-some-key-areas-the-way-ldb-mo.patch
> > > > 
> > > > Also git tree here: git://git.samba.org/idra/samba.git
> > > > branch v4-0-ldb-async
> > > 
> > > - search request callbacks MUST not issue new requests until the
> > >   LDB_REPLY_DONE ldb_reply is received. Once this reply is received a
> > >   module can issue a new request or return to the upper module using
> > >   ldb_module_done()
> > > 
> > > 
> > > Why do you want this restriction?  In a number of modules, this pattern
> > > has allowed an easy, asynchronous handling of some difficult situations
> > > (such as handling subtree renames, without building massive lists in
> > > memory). 
> > 
> > Which only worked by chance
> 
> It seemed to work perfectly fine in ldb_tdb and against LDAP...

Yes probably of a side effect of calling ldb_wait and intercepting it it
was working even with LDAP, but trust me it was *really* dirty.

> >  because ldb_tdb ws async and giving back
> > results before the former request was done (this does not happen anymore
> > now even with ldb_tdb).
> > 
> > It simply does not work.
> 
> Why not?  I'm confused - it seems a very natural pattern to 'do
> something' in the callback, where that something may well be a
> modification to the records.
> 
> Why should this not be possible?  How do you suggest implementing
> 'iterate and fix' modules?

See ldb_map I collect the data and the fire a new request when I get
LDB_REPLY_DONE, repeat until done.

This also helps getting just one outstanding call at a time, with the
previous method you could end up with countless outstanding calls and
some ldap server has limitations on the number of calls each single
client can have outstanding, so the previous way may also lead to
unpredictable results.

Simo.

-- 
Simo Sorce
Samba Team GPL Compliance Officer <simo at samba.org>
Senior Software Engineer at Red Hat Inc. <ssorce at redhat.com>