status of DRS efforts in Samba4 (and a developer tutorial)

[Nadezhda Ivanova]

Hi all,
Tridge's todo list is now on the wiki at
http://wiki.samba.org/index.php/Samba4_DRS_TODO_List

----- Original Message -----
> From: samba-technical-bounces at lists.samba.org <samba-technical-bounces at lists.samba.org>
> To: samba-technical at samba.org <samba-technical at samba.org>, tridge at samba.org <tridge at samba.org>
> Cc: Rodolfo Jardim de Azevedo <rodolfo at ic.unicamp.br>, abartlet at samba.org <abartlet at samba.org>, Fernando Silva <fernandojvsilva at yahoo.com.br>
> Sent: Saturday, September 19, 2009 3:08:23 PM GMT-0800 America;Los_Angeles
> Subject: Re: status of DRS efforts in Samba4 (and a developer tutorial)

> > This is a quick list of some of the more obvious 'todo' items for the
> DRS implementation in Samba4. I have included a brief explanation of
> each item, but please ask if you want more detail.
> 
> The idea of this list is to give people somewhere to start if they are
> looking for something to contribute.
> 
> ---------------------------------------------
> - join w2k8 to samba4 dc
> 
> We've been concentrating up to now on Samba4<->Samba4 replication, and
> Samba4<->Windows replication where the Samba4 server joins the Windows
> domain. A more difficult problem is making it work when you start with
> a Samba4 domain (from provision, or from vampiring a Windows domain)
> and then try to add another Windows DC by using dcpromo. This is
> currently failing with an obscure error at the end of the dcpromo
> process.
> 
> ---------------------------------------------
> - create connection object (nTDSConnection)
> 
> Our KCC implementation (in source4/dsdb/kcc) is very simple at the
> moment. It should work by creating nTDSConnection objects under the
> nTDSDSA objects in the LDAP tree, then use those to create the
> repsFrom attributes, and possibly send DsUpdateRefs operations to the
> other DCs to setup a repsTo on each replication partner.
> 
> Right now we don't create nTDSConnection objects at all, which needs
> to be fixed.
> 
> ---------------------------------------------
> - update to new doc release
> 
> We should look through the new WSPP docs release (from August 2009)
> and see what we haven't implemented yet, forming a more extensive todo
> list then this one. Now that we have basic replication working we can
> start to try to get all the corner cases right, and for that the docs
> (especially MS-DRSR and MS-ADTS) are a good source of information.
> 
> ---------------------------------------------
> - why isn't repsTo written by Windows?
> 
> I have noticed that Windows is not sending us a DsUpdateRefs to update
> the repsTo when we join a Windows domain as a 2nd DC. This means if we
> followed the correct behaviour we would never send Windows a
> DsReplicaSync message, so we'd never tell windows to replication to
> us.
> 
> To work around this dreplsrv_notify_check() currently cheats by using
> repsFrom if repsTo is empty. We need to instead work out why Windows
> is not sending us DsUpdateRefs messages. Perhaps related to the lack
> of nTDSConnection objects?
> 
> ---------------------------------------------
> - repadmin.exe tool
> 
> The repadmin.exe tool on windows is a great way of seeing the status
> of replication. We would like to get all of the options of repadmin
> working when directed at a Samba4 DC. Anatoliy is working on making
> some of the fnuctions work, but there are plenty more to do.
> 
> ---------------------------------------------
> - hook delete in repl_meta_data
> 
> Right now we just pass delete operations down through the
> repl_meta_data module to the ldb_tdb backend. That means that deletes
> are not replicated (as they don't change anything in
> ReplPropertyMetaData or in the uSNChanged attribute).
> 
> We should intercept delete operations and translate them into a
> combination of a rename to a objected in the "Deleted Objects" tree,
> along with a modify to add the isDeleted attribute. Then we need to
> setup the tombstone data in the object, and add a tombstone reaping
> task that would run once a day to really delete expired tombstone
> records.
> 
> ---------------------------------------------
> - phantom objects
> 
> If DRS replication adds a link to a object that doesn't exist we are
> supposed to create a "phantom" object, which gets filled in later. We
> are working around that at the moment by delating link creation until
> then end of the transaction for the replica cycle, but we should also
> support phantom objects.
> 
> ---------------------------------------------
> - sort objects on disk
> 
> Some sysadmins might write scripts that rely on the return order of
> attributes within objects (eg. objectclass first). We sort objects on
> add in repl_meta_data.c to cope with this but we don't fix the sorting
> on modify. That should be fixed.
> 
> ---------------------------------------------
> - speed up replmd_ldb_message_element_attid_sort
> 
> The replmd_ldb_message_element_attid_sort function is pretty
> inefficient. We need to avoid the attribute lookups in the sort
> comparison function.
> 
> ---------------------------------------------
> - don't allow replication of readonly attribs
> 
> We should not allow replication to overwrite readonly
> attributes. There are other attribute flags we aren't honouring as
> well. We should check the docs and add support for all the relevent
> attribute flags.
> 
> ---------------------------------------------
> - support RODC
> 
> A RODC (read-only domain controller) is a potentially very useful use
> case for Samba4. There is quite a lot of changes in replication and
> attribute filtering that shoud be done when we are a RODC.
> 
> ---------------------------------------------
> - separate gc partition
> 
> Right now the gc partition is just an amalgamation of the normal base
> partitions, with no filtering (we just set the magic control to say
> that searches should cross partition boundaries).
> 
> We need to decide if we should make a separate ldb for the gc
> partition, and if so what method we will use to keep it in sync. If we
> don't create a separate partition then we should add the right
> filtering to gc searches.
> 
> ---------------------------------------------
> - if modify sets attrib to same value then no replPropertyMetaData 
> change
> 
> A modify via DRS replication that asks for an attribute to change to
> the same value it already has should be filtered out by
> repl_meta_data.c so that the replPropertyMetaData attribute is not
> updated.
> 
> ---------------------------------------------
> - fix error mapping (no FOOBAR, and replmd_replicated_request_werror)
> 
> We have lots of code that returns WERR_FOOBAR or NT_STATUS_FOOBAR
> because we didn't know what error to return. We need to go through
> these and either work out the correct error code, or if that is hard
> then at least put a reasonable guess of the right error code along
> with a TODO comment to check it.
> 
> ---------------------------------------------
> - parentguid fix
> 
> We store parentGUID in the object on disk at the moment, whereas we
> should construct it at runtime when asked for.
> 
> ---------------------------------------------
> - honor attribute replication flag
> 
> There is a attribute flag for whether particular attributes should be
> replicated. We need to check that we get this right.
> 
> ---------------------------------------------
> - double cn fix
> 
> When we do a s4<->s4 vampire we end up with the rDN attribute
> appearing twice on all objects in the new replica. We think this is
> because we should be filtering the rDN in the getncchanges code, but
> this needs checking.
> 
> ---------------------------------------------
> - check for parent exists in replication add and rename
> 
> during replication add and rename we need to check that the
> destination parent exists.
> 
> ---------------------------------------------
> - handle add where DN exists, but different GUID
> 
> We may need to handle the case where a DRS replication comes in for a
> DN that exists, but with a different GUID. We need to test with
> windows on how this is handled.
> 
> ---------------------------------------------
> - net commands to query repl status (via DRS?)
> 
> We should add net commands for querying the replication status
> (somewhat like repadmin.exe).
> 
> ---------------------------------------------
> - max number of attributes on objects?
> 
> Metze noticed that the WSPP docs specify a maximum object size in AD
> of around 8k. This seems to translate into a maximum number of
> attributes that windows accepts. We may need to implement a similar
> limit to prevent problems with replication s4->windows.
> 
> ---------------------------------------------
> - obey acls on objects
> 
> We need to obey the ntSecurityDescriptor on objects in our SAM. This
> is a large task! Nadia is working on it and hopefully will merge soon.
> 
> ---------------------------------------------
> - fix ldb_add ojbectclass sorting
> 
> In ldb_add we sort objectClass attributes in the objectclass
> module. The sort is currently horribly inefficient - it needs redoing
> using the sort indexes that Andrew and Nadia have recently added.
> 
> ---------------------------------------------
> - -s option to setup_dns.sh
> 
> The setup_dns.sh should be redone as a python wrapper so it obeys
> standard options like -s and can read smb.conf
> 
> ---------------------------------------------
> - what triggers initial kcc run on windows after we join a w2k8 DC?
> 
> After we join a s4 DC to a windows domain, we've noticed that w2k8
> needs to be prompted to run its KCC using "repadmin /kcc". We need to
> work out why this is needed so we can fix it.
> 
> ---------------------------------------------
> - s4<->s4 in make test
> 
> We should add the s4<->s4 vampire and replication in make test
> 
> ---------------------------------------------
> - urgent replication
> 
> We need to add the urgent bit on replications that have changed
> critical objects (see the docs for a list). We will probably need to
> expand @REPLCHANGED to add a uSNUrgent attribute to support this.
> 
> ---------------------------------------------
> - group policies
> 
> We are not currently obeying group policies, although we can serve
> them out to clients. We need to obey the ones that make sense for
> Samba. For this we need to provide a really easy API to allow any part
> of Samba to query a group policy, and to auto-update SAMDB with the
> needed changes.
> 
> ---------------------------------------------
> - linked attribs
> 
> We currently accept the w2k8 linked attributes in replication, but
> when other DCs replicate to us we serve up linked attributes as normal
> attributes (which is like a downlevel w2k3 does). We should store the
> full meta data associated with linked attributes in more fields in the
> extended DN and serve it up in getncchanges.
> 
> ---------------------------------------------
> - add support for ndr64 to wireshark
> 
> When watching w2k8-R2 <-> w2k8-R2 interactions, windows chooses
> NDR64 instead of NDR. We now support NDR64 in Samba, but wireshark
> doesn't understand it. To allow us to watch traffic between w2k8-R2
> boxes we would like wireshark to understand NDR64.
> 
> ---------------------------------------------
> - convert wireshark drsuapi to pidl
> 
> The DRSUAPI decoder in wireshark is quite poor. We should redo it
> using a pidl based parser.
> 
> ---------------------------------------------
> - fix decryption of w2k8 by wireshark (krb5 patch)
> 
> Wehn watching w2k8 <-> samba traffic in wireshark we often find that
> wireshark cannot decrypt some of the traffic. This is due to a
> bug/limitation in MIT kerberos. Metze has a hack based on LD_PRELOAD
> that works around this, but we should try to get this into the
> wireshark svn tree directly.
> 
> ---------------------------------------------
> - bitmap32 actually 3264 in samr QueryUserInfo level 16? (netmon bug
>   too)
> 
> There seems to be a problem with the QueryUserInfo level 16 and
> NDR64. The Microsoft netmon 3.3 parser has the same problem as our
> ndrdump parser. We need to look into how this should be handled.
> 
> ---------------------------------------------
> - how does another DC become the FSMO master and RID master
> 
> We need to work out how a DC should become the FSMO master and RID
> master. We can do it now via ldbedit, but there should be a more
> automated method (perhaps the KCC should do this?)
> 
> Nadia, you mentioned that you might put this up on the Samba wiki? It
> would be good to tick these off as we get them done.
> 
> Cheers, Tridge