[PATCH] s4-drs: replmd_delete implementation

[tridge at samba.org]

Hi Eduardo,

 > I created this utility function that gets the defaultNamingContext and put
 > "CN=Deleted Objects" in front of the DN.

I found your git tree at git://repo.or.cz/Samba/eduardoll.git and had
a look at the function. It has a couple of problems:

 1) it suffers from the very common cut&paste problem with error
 messages - I think 'dsServiceName' in the error message should be
 'defaultNamingContext'

 2) it doesn't add the "Deleted Objects" rDN to the returned
 result. You could use ldb_dn_add_child_fmt() to add it like this:

   deleted_dn = ldb_dn_add_child_fmt(base_dn, "Deleted Objects");

 3) I don't think you actually need to do the search for
 defaultNamingContext. Take a look at the function
 samdb_partitions_dn() for what would be a closer example to what you
 need. That one adds "CN=Partitions" to samdb_config_dn(). If you
 instead add "CN=Deleted Objects" to samdb_base_dn() then I think
 you'll have what you want.

 > Will this work for every type of object? (every object in the
 > domain - even if it's in the deepest depth - should be moved
 > directly to CN=Deleted Objects,DC=w2k8... when deleted?)

hmm, looking at a w2k8 server, it looks like there is a separate
"Deleted Objects" container in the configuration partition. That means
my suggestion above is wrong, and you'll instead need to work out the
correct "Deleted Objects" location based upon the DN of the object
being deleted.

One curious thing is that there is no "Deleted Objects" container in
the schema partition on w2k8. I think you should investigate what
happens to deleted objects in the schema partition. Perhaps if a
partition does not have a Deleted Objects container then
the repl_meta_data module should not do the rename/modify change and
instead should just delete it? Or maybe it get moved into the Deleted
Objects container of the next partition up?

You should investigate this by creating and then deleting an object in
a schema partitino of a w2k8 server and see what happens to the
object. If you run something like this:

  bin/ldbsearch -H ldap://w2k8 -Uadministrator%password --controls show_deleted:1,search_options:1:2 'isDeleted=TRUE' dn objectclass lastKnownParent

then it will give you an idea of where the deleted objects are going.

Once you've done that, you'll need to add a function in
repl_meta_data.c that works out the DN of the Deleted Objects
container given the DN of an object being deleted. There are several
ways to do this. One reasonable approach would be to add a function in
dsdb/common/util.c like this:

  struct ldb_dn *dsdb_find_nc_root(struct ldb_context *samdb, struct ldb_dn *dn);

it would work by chopping off components from the DN (using
ldb_dn_get_parent()) one at a time until it found a DN that has an
instanceType with INSTANCE_TYPE_IS_NC_HEAD set. You'll need to do a
base search for instanceType for each DN as you loop.

Then you'd add the "CN=Deleted Objects" to that, and see if it
exists. If it does then you can do the rename/modify. If not then just
pass the original delete request down to the next module.

You will also need to change our provision to add a "Deleted Objects"
container for the configuration partition. If you look at
setup/provision.ldif you'll see that it has adds "Deleted Objects"
container for the DOMAINDN, but not for the configuration
partition. You'll need to add it to setup/provision_configuration_basedn.ldif.

 > This is the part that I'm having more difficulty. I still can't understand
 > well what the modules are and the way that they work in the Samba source.  I
 > saw that ldb_module structure has *prev and *next fields and modules might
 > be linked together but why would all the modules be renamed if I use the
 > ldb_rename with the top level ldb context? When I do a bin/ldbdel operation,
 > what is the execution's flow that the modules follow?

This is one of the more complex aspects of Samba. I'll try and explain
it, and if you have any more questions then please ask.

Plain ldb is very simple. It knows nothing about the complexities of
active directory. It knows nothing about schemas, or objectclasses or
any of the things that go to make up a AD compatible system. It just
knows how to store, retrieve and modify objects in a dumb fashion.

The way it becomes more like AD is to have a series of modules. If you
do this search:

  bin/ldbsearch -H $PREFIX/private/sam.ldb -b @MODULES -s base

then you'll see the list of modules that are in your database. That
list currently looks like this:

@LIST: resolve_oids,rootdse,lazy_commit,paged_results,ranged_results,anr,server_sort,asq,extended_dn_store,extended_dn_in,rdn_name,objectclass,descriptor,acl,samldb,password_hash,operational,kludge_acl,instancetype,repl_meta_data,subtree_rename,subtree_delete,linked_attributes,extended_dn_out_ldb,show_deleted,schema_load,new_partition,partition

as you can see, there are a lot of modules!

When you call an ldb operation, for example a ldb_delete() call, then
the ldb code will check with each of these modules in the order they
are listed, and if the module has a registered 'delete' function then
that function is called.

If you look at the first module in the list (the resolve_oids module)
and look near the bottom of resolve_oids.c you'll see that it doesn't
have a function for delete. That means that the resolve_oids module
will not affect delete operations.

It does however have a method for 'add'. So let's look at what that
does.

Imagine we called ldb_add() to add an object to the database. The ldb
code will see that the resolve_oids module has an 'add' method, so it
will call resolve_oids_add(). If you look inside that function you'll
see that it can complete in a number of ways:

 1) if it decides that it doesn't need to do anything, it can call
 ldb_next_request() passing the original request (the 'req'
 argument). That asks ldb to call the next module in the list that has
 an 'add' method.

 2) it can return an error. You can see for examples places where it
 returns LDB_ERR_OPERATIONS_ERROR when it fails to allocate some
 memory.

 3) it can construct a new request which does something different, and
 then call ldb_next_request() with that new request. This is what it
 does in this line:

	return ldb_next_request(module, down_req);

 That asks ldb to pass a newly constructed request down to the next
 module. Notice that it has specified a callback
 (resolve_oids_callback) that should be called when this new request
 is finished.

In this way, each ldb request passes through each of the modules that
wants to be involved with that type of request. Eventually the request
gets down to the last module in the chain, which is the ldb_tdb module
that actually implements the backend (that is not shown in the @LIST
above, it is implied). The ldb_tdb module, which just implements dumb
message storage, does the actual change to the database.

So, back to my original comment on using ldb_rename() with a ldb
context. The way you had done it will work, but it probably isn't
quite the right thing to do. You are calling it from within the
repl_meta_data module. That means you are half way through the list of
modules above. All of the modules above have either done what they
want to do to this request, or don't want to do anything.

By calling ldb_rename() you are starting at the top of the module list
again. This means every module above you will see the rename. Do you
want that? I think in this case you probably don't, and what you
really want to do is pass the rename down to the modules below you,
skipping the ones above.

One of the reasons we try to do this is to avoid looping forever. If
we go to the top of the stack for this rename, then that means the
rename will come back down to the repl_meta_data module again. So
we'll be calling our own module. I think in this case it will work,
but it can be a bit hard to follow sometimes!

If you look at dsdb/samdb/ldb_modules/util.c then you can see some
helper functions have been added to make this sort of "do an operation
on the rest of the modules" call a bit easier. The helper functions
are currently only for search, but you could add a new one for doing a
rename starting at the current module. Following the pattern in that
file, you'd add a new function:

 int dsdb_module_rename(struct ldb_module *module,
 		        struct ldb_dn *olddn, struct ldb_dn *newdn);

it would call ldb_build_rename_req() to build a new rename request
structure, then it would call ldb_next_request() and then call
ldb_wait(), in much the same way that dsdb_module_search() works.

If you then use dsdb_module_rename() in your new code, then you will
be doing the rename only on the modules below you in the stack. You
should similarly add a dsdb_module_modify() call that follows the same
pattern.

The only caveat to this is if there is a module above you in the stack
that you actually want to see the rename. For example, the rdn_name
module (in lib/ldb/modules/rdn_name.c) has special handling for rename
operations to fix the 'name' field. If you follow my suggestion above
then this special handling won't happen. What you need to do is check
on a w2k8 server and see if the 'name' attribute is changed when you
delete an object.

If I do this:

  bin/ldbsearch -H ldap://w2k8 -Uadministrator%password --controls show_deleted:1,search_options:1:2 'isDeleted=TRUE' dn objectclass lastKnownParent name --show-binary

then I can see that, yes, the 'name' attribute does look like it is
being modified when you do a delete. That means we have two choices:

  1) also fix the 'name' attribute in the modify operation that you
  will do as part of the delete handling in repl_meta_data.c. Add it
  to the same modify that adds the isDeleted=TRUE attribute.

  2) forget what I said above and just use ldb_rename(). That means
  that the rdn_name modules will do the fixup of 'name' for you.

I think that (1) is a better choice, as I think it makes it clearer
exactly what a delete does in the repl_meta_data module. 

I hope this helps!

Cheers, Tridge