status of DRS efforts in Samba4 (and a developer tutorial)

[tridge at samba.org]

This is a quick list of some of the more obvious 'todo' items for the
DRS implementation in Samba4. I have included a brief explanation of
each item, but please ask if you want more detail.

The idea of this list is to give people somewhere to start if they are
looking for something to contribute.

---------------------------------------------
- join w2k8 to samba4 dc

We've been concentrating up to now on Samba4<->Samba4 replication, and
Samba4<->Windows replication where the Samba4 server joins the Windows
domain. A more difficult problem is making it work when you start with
a Samba4 domain (from provision, or from vampiring a Windows domain)
and then try to add another Windows DC by using dcpromo. This is
currently failing with an obscure error at the end of the dcpromo
process.

---------------------------------------------
- create connection object (nTDSConnection)

Our KCC implementation (in source4/dsdb/kcc) is very simple at the
moment. It should work by creating nTDSConnection objects under the
nTDSDSA objects in the LDAP tree, then use those to create the
repsFrom attributes, and possibly send DsUpdateRefs operations to the
other DCs to setup a repsTo on each replication partner.

Right now we don't create nTDSConnection objects at all, which needs
to be fixed.

---------------------------------------------
- update to new doc release

We should look through the new WSPP docs release (from August 2009)
and see what we haven't implemented yet, forming a more extensive todo
list then this one. Now that we have basic replication working we can
start to try to get all the corner cases right, and for that the docs
(especially MS-DRSR and MS-ADTS) are a good source of information.

---------------------------------------------
- why isn't repsTo written by Windows?

I have noticed that Windows is not sending us a DsUpdateRefs to update
the repsTo when we join a Windows domain as a 2nd DC. This means if we
followed the correct behaviour we would never send Windows a
DsReplicaSync message, so we'd never tell windows to replication to
us.

To work around this dreplsrv_notify_check() currently cheats by using
repsFrom if repsTo is empty. We need to instead work out why Windows
is not sending us DsUpdateRefs messages. Perhaps related to the lack
of nTDSConnection objects?

---------------------------------------------
- repadmin.exe tool

The repadmin.exe tool on windows is a great way of seeing the status
of replication. We would like to get all of the options of repadmin
working when directed at a Samba4 DC. Anatoliy is working on making
some of the fnuctions work, but there are plenty more to do.

---------------------------------------------
- hook delete in repl_meta_data

Right now we just pass delete operations down through the
repl_meta_data module to the ldb_tdb backend. That means that deletes
are not replicated (as they don't change anything in
ReplPropertyMetaData or in the uSNChanged attribute).

We should intercept delete operations and translate them into a
combination of a rename to a objected in the "Deleted Objects" tree,
along with a modify to add the isDeleted attribute. Then we need to
setup the tombstone data in the object, and add a tombstone reaping
task that would run once a day to really delete expired tombstone
records.

---------------------------------------------
- phantom objects

If DRS replication adds a link to a object that doesn't exist we are
supposed to create a "phantom" object, which gets filled in later. We
are working around that at the moment by delating link creation until
then end of the transaction for the replica cycle, but we should also
support phantom objects.

---------------------------------------------
- sort objects on disk

Some sysadmins might write scripts that rely on the return order of
attributes within objects (eg. objectclass first). We sort objects on
add in repl_meta_data.c to cope with this but we don't fix the sorting
on modify. That should be fixed.

---------------------------------------------
- speed up replmd_ldb_message_element_attid_sort

The replmd_ldb_message_element_attid_sort function is pretty
inefficient. We need to avoid the attribute lookups in the sort
comparison function.

---------------------------------------------
- don't allow replication of readonly attribs

We should not allow replication to overwrite readonly
attributes. There are other attribute flags we aren't honouring as
well. We should check the docs and add support for all the relevent
attribute flags.

---------------------------------------------
- support RODC

A RODC (read-only domain controller) is a potentially very useful use
case for Samba4. There is quite a lot of changes in replication and
attribute filtering that shoud be done when we are a RODC.

---------------------------------------------
- separate gc partition

Right now the gc partition is just an amalgamation of the normal base
partitions, with no filtering (we just set the magic control to say
that searches should cross partition boundaries).

We need to decide if we should make a separate ldb for the gc
partition, and if so what method we will use to keep it in sync. If we
don't create a separate partition then we should add the right
filtering to gc searches.

---------------------------------------------
- if modify sets attrib to same value then no replPropertyMetaData change

A modify via DRS replication that asks for an attribute to change to
the same value it already has should be filtered out by
repl_meta_data.c so that the replPropertyMetaData attribute is not
updated.

---------------------------------------------
- fix error mapping (no FOOBAR, and replmd_replicated_request_werror)

We have lots of code that returns WERR_FOOBAR or NT_STATUS_FOOBAR
because we didn't know what error to return. We need to go through
these and either work out the correct error code, or if that is hard
then at least put a reasonable guess of the right error code along
with a TODO comment to check it.

---------------------------------------------
- parentguid fix

We store parentGUID in the object on disk at the moment, whereas we
should construct it at runtime when asked for.

---------------------------------------------
- honor attribute replication flag

There is a attribute flag for whether particular attributes should be
replicated. We need to check that we get this right.

---------------------------------------------
- double cn fix

When we do a s4<->s4 vampire we end up with the rDN attribute
appearing twice on all objects in the new replica. We think this is
because we should be filtering the rDN in the getncchanges code, but
this needs checking.

---------------------------------------------
- check for parent exists in replication add and rename

during replication add and rename we need to check that the
destination parent exists.

---------------------------------------------
- handle add where DN exists, but different GUID

We may need to handle the case where a DRS replication comes in for a
DN that exists, but with a different GUID. We need to test with
windows on how this is handled.

---------------------------------------------
- net commands to query repl status (via DRS?)

We should add net commands for querying the replication status
(somewhat like repadmin.exe).

---------------------------------------------
- max number of attributes on objects?

Metze noticed that the WSPP docs specify a maximum object size in AD
of around 8k. This seems to translate into a maximum number of
attributes that windows accepts. We may need to implement a similar
limit to prevent problems with replication s4->windows.

---------------------------------------------
- obey acls on objects

We need to obey the ntSecurityDescriptor on objects in our SAM. This
is a large task! Nadia is working on it and hopefully will merge soon.

---------------------------------------------
- fix ldb_add ojbectclass sorting

In ldb_add we sort objectClass attributes in the objectclass
module. The sort is currently horribly inefficient - it needs redoing
using the sort indexes that Andrew and Nadia have recently added.

---------------------------------------------
- -s option to setup_dns.sh

The setup_dns.sh should be redone as a python wrapper so it obeys
standard options like -s and can read smb.conf

---------------------------------------------
- what triggers initial kcc run on windows after we join a w2k8 DC?

After we join a s4 DC to a windows domain, we've noticed that w2k8
needs to be prompted to run its KCC using "repadmin /kcc". We need to
work out why this is needed so we can fix it.

---------------------------------------------
- s4<->s4 in make test

We should add the s4<->s4 vampire and replication in make test

---------------------------------------------
- urgent replication

We need to add the urgent bit on replications that have changed
critical objects (see the docs for a list). We will probably need to
expand @REPLCHANGED to add a uSNUrgent attribute to support this.

---------------------------------------------
- group policies

We are not currently obeying group policies, although we can serve
them out to clients. We need to obey the ones that make sense for
Samba. For this we need to provide a really easy API to allow any part
of Samba to query a group policy, and to auto-update SAMDB with the
needed changes.

---------------------------------------------
- linked attribs

We currently accept the w2k8 linked attributes in replication, but
when other DCs replicate to us we serve up linked attributes as normal
attributes (which is like a downlevel w2k3 does). We should store the
full meta data associated with linked attributes in more fields in the
extended DN and serve it up in getncchanges.

---------------------------------------------
- add support for ndr64 to wireshark

When watching w2k8-R2 <-> w2k8-R2 interactions, windows chooses
NDR64 instead of NDR. We now support NDR64 in Samba, but wireshark
doesn't understand it. To allow us to watch traffic between w2k8-R2
boxes we would like wireshark to understand NDR64.

---------------------------------------------
- convert wireshark drsuapi to pidl

The DRSUAPI decoder in wireshark is quite poor. We should redo it
using a pidl based parser.

---------------------------------------------
- fix decryption of w2k8 by wireshark (krb5 patch)

Wehn watching w2k8 <-> samba traffic in wireshark we often find that
wireshark cannot decrypt some of the traffic. This is due to a
bug/limitation in MIT kerberos. Metze has a hack based on LD_PRELOAD
that works around this, but we should try to get this into the
wireshark svn tree directly.

---------------------------------------------
- bitmap32 actually 3264 in samr QueryUserInfo level 16? (netmon bug
  too)

There seems to be a problem with the QueryUserInfo level 16 and
NDR64. The Microsoft netmon 3.3 parser has the same problem as our
ndrdump parser. We need to look into how this should be handled.

---------------------------------------------
- how does another DC become the FSMO master and RID master

We need to work out how a DC should become the FSMO master and RID
master. We can do it now via ldbedit, but there should be a more
automated method (perhaps the KCC should do this?)

Nadia, you mentioned that you might put this up on the Samba wiki? It
would be good to tick these off as we get them done.

Cheers, Tridge