svn commit: samba r10875 - in branches/tmp/samba4-winsrepl: . source/include source/libcli/util source/librpc/idl source/librpc/ndr source/pidl/lib/Parse/Pidl/Samba/NDR source/torture source/torture/rpc

Mon Oct 10 15:12:57 GMT 2005

Author: metze
Date: 2005-10-10 15:12:56 +0000 (Mon, 10 Oct 2005)
New Revision: 10875

WebSVN: http://websvn.samba.org/cgi-bin/viewcvs.cgi?view=rev&root=samba&rev=10875

Log:
 r13572 at SERNOX (orig r10865):  metze | 2005-10-10 11:33:06 +0200
 merge branches/SOC/SAMBA_4_0 into main the main SAMBA_4_0 tree

 metze

  r8017 at SERNOX:  metze | 2005-06-30 13:44:23 +0200
  create the SAMBA_4_0 branch for the Summer Of Code Project

  metze

  r8730 at SERNOX:  brad | 2005-07-24 03:09:48 +0200
  Branching Samba 4
  r8731 at SERNOX:  brad | 2005-07-24 06:39:00 +0200
  added 'make installmisc' to howto.txt
  added existing 'compression' option to level8 drsuapi torture test
  added new 'neighbour_writeable' option to level8 drsuapi torture test
  r8732 at SERNOX:  brad | 2005-07-24 06:42:38 +0200
  added metze's dssync patch as source/torture/rpc/dssync.c
  r8739 at SERNOX:  brad | 2005-07-25 00:24:46 +0200
  added a test called RPC-DSSYNC to config.mk
  hacking at dssync.c in an attempt to make it compile
  r8754 at SERNOX:  brad | 2005-07-25 15:19:21 +0200
  Changing dssync.c to use ldb routines for accessing ldap rather than raw ldap calls.

  r8765 at SERNOX:  brad | 2005-07-26 03:35:38 +0200
  more ldb changes to test_CompleteJoin(), it mostly kind of almost works now!

  r8766 at SERNOX:  brad | 2005-07-26 03:56:00 +0200
  Trying to fix the crazy nesting in the branch
  r8769 at SERNOX:  brad | 2005-07-26 04:48:29 +0200
  merging latest changes
  r8770 at SERNOX:  brad | 2005-07-26 04:53:43 +0200
  removing nested branch
  r8793 at SERNOX:  jerry | 2005-07-27 05:04:57 +0200
  merging on of Brad missing changes from the nested 4.0 branch debacle
  r8794 at SERNOX:  jerry | 2005-07-27 05:14:42 +0200
  syncing up with the main 4_0 branch for Brad
  r8842 at SERNOX:  brad | 2005-07-29 00:26:30 +0200
  merging changes from branches/SAMBA_4_0
  r8850 at SERNOX:  brad | 2005-07-29 21:07:57 +0200
  Bringing my tree up to date
  r8851 at SERNOX:  brad | 2005-07-30 00:48:04 +0200
  making dssync.c more ldb-centric, reverted samlogon.c from rev. 8845 to get my branch to compile again.
  r8856 at SERNOX:  brad | 2005-07-30 03:20:33 +0200
  I think I have the ldb code down in test_CompleteJoin (not complete yet though)
  r8860 at SERNOX:  brad | 2005-07-30 07:08:13 +0200
  Changed comments to C style /**/ (thanks Richard), some more changes to test_CompleteJoin(). 
  r8862 at SERNOX:  brad | 2005-07-31 04:45:32 +0200
  Bringing the SOC/SAMBA_4_0 branch up to date.
  r8863 at SERNOX:  brad | 2005-07-31 20:00:41 +0200
  Updated some missing files from the branch
  r8864 at SERNOX:  brad | 2005-07-31 20:25:50 +0200
  Removing autogenerated files from branch
  r8865 at SERNOX:  brad | 2005-07-31 20:43:58 +0200
  last of the unneeded files in SOC/SAMBA_4_0
  r9004 at SERNOX:  brad | 2005-08-03 18:51:23 +0200
   r5214 at buttercup:  j0j0 | 2005-08-03 10:44:30 -0600
    r at buttercup:  j0j0 | 2005-08-02 22:54:13 -0600
    creating a local branch of branches/SAMBA_4_0

  r9013 at SERNOX:  brad | 2005-08-03 20:57:48 +0200
   r5228 at buttercup:  j0j0 | 2005-08-03 13:00:11 -0600
   Fixing differences between this branch and /branches/SAMBA_4_0

  r9014 at SERNOX:  brad | 2005-08-03 21:18:05 +0200
   r5231 at buttercup:  j0j0 | 2005-08-03 13:23:12 -0600
   Updating config.mk so that smbtorture builds again

  r9061 at SERNOX:  brad | 2005-08-04 18:17:36 +0200
   r5249 at buttercup:  j0j0 | 2005-08-03 21:01:02 -0600
   Start using libnet_Join() for DC join.

  r9062 at SERNOX:  brad | 2005-08-04 18:17:47 +0200
   r5250 at buttercup:  j0j0 | 2005-08-04 10:21:34 -0600
   Some more work towards performing a dc join.

  r9064 at SERNOX:  brad | 2005-08-04 18:53:51 +0200
   r5253 at buttercup:  j0j0 | 2005-08-04 10:53:00 -0600
   Fixed a bug (passing a TALLOC_CTX to libnet_context_init() )

  r9069 at SERNOX:  brad | 2005-08-04 21:59:55 +0200
   r5279 at buttercup:  j0j0 | 2005-08-04 14:04:55 -0600
   Some more work on the domain join

  r9117 at SERNOX:  brad | 2005-08-05 16:50:26 +0200
   r5281 at buttercup:  j0j0 | 2005-08-05 08:55:58 -0600
   Committing minor changes before merge

  r9180 at SERNOX:  brad | 2005-08-07 17:25:25 +0200
   r5314 at buttercup:  j0j0 | 2005-08-07 09:30:12 -0600
   Reworked libnet_join to use two join levels, AUTOMATIC and SPECIFIED.

  r9181 at SERNOX:  brad | 2005-08-07 17:25:36 +0200
   r5315 at buttercup:  j0j0 | 2005-08-07 09:31:22 -0600
   Working with libnet_Join(), code cleanup needed in the near future.

  r9192 at SERNOX:  brad | 2005-08-07 21:40:22 +0200
   r5373 at buttercup:  j0j0 | 2005-08-07 13:46:09 -0600
   Some code cleanup to make things a little more readable.

  r9249 at SERNOX:  brad | 2005-08-12 01:31:48 +0200
   r5375 at buttercup:  j0j0 | 2005-08-11 17:38:44 -0600
   Split libnet_JoinDomain() into libnet_JoinDomain() and libnet_JoinADSDomain().

  r9256 at SERNOX:  brad | 2005-08-12 04:55:11 +0200
   r5413 at buttercup:  j0j0 | 2005-08-11 21:02:27 -0600
   Clean up libnet_JoinADSDomain() a little, added a comment to the test_join struct.

  r9314 at SERNOX:  brad | 2005-08-16 03:53:20 +0200
   r5436 at buttercup:  j0j0 | 2005-08-15 20:01:21 -0600
   libnet_JoinDomain() should honour LIBNET_JOIN_TORTURE now. 
   torture_join_domain() should properly use libnet_JoinDomain().
   dssync.c uses torture_join_domain() again.

  r9351 at SERNOX:  brad | 2005-08-17 07:15:31 +0200
   r5438 at buttercup:  j0j0 | 2005-08-16 23:23:58 -0600
   Removed LIBNET_JOIN_TORTURE level, as it became unnecessary once libnet_Join_primary_domain() handled netbios names better.
   Corrected libnet_JoinDomain() and libnet_JoinADSDomain().

  r9352 at SERNOX:  brad | 2005-08-17 07:24:49 +0200
   r5440 at buttercup:  j0j0 | 2005-08-16 23:33:25 -0600
   Fixed a typo.

  r9354 at SERNOX:  metze | 2005-08-17 10:28:25 +0200
  remove object files from svn

  metze
  r9376 at SERNOX:  brad | 2005-08-18 05:15:48 +0200
   r5476 at buttercup:  j0j0 | 2005-08-17 21:24:33 -0600
   Proof that I shouldn't code when i'm tired (silly bugfixes).

  r9405 at SERNOX:  brad | 2005-08-19 22:50:10 +0200
   r5500 at buttercup:  j0j0 | 2005-08-19 14:56:25 -0600
   Get dssync.c compiling again after merge (ldb_dn changes from rev. 9391).

  r9407 at SERNOX:  brad | 2005-08-20 03:22:42 +0200
   r5502 at buttercup:  j0j0 | 2005-08-19 19:28:22 -0600
   libnet/libnet_join.c
   Some more fixes so ldb uses ldb_dn's.

   torture/rpc/dssync.c
   Some debugging printf()'s.
   ldb_dn fixes.

   torture/rpc/testjoin.c
   Change torture_join_domain() to use libnet_JoinDomain() rather than libnet_Join().
   Some more debugging statements.

   I'm not sure why, but GUID_all_zero(user_handle.uuid) is returning true in torture_leave_domain() when called it from torture_destroy_context() in torture/rpc/dssync.c.
   That's what i'm working out now.

  r9427 at SERNOX:  brad | 2005-08-20 18:38:29 +0200
   r5504 at buttercup:  j0j0 | 2005-08-20 10:44:52 -0600
   Some bugfixes.
   Removed a bunch of debugging code.
   torture_leave_domain() works again! not 100% perfect yet though...

  r9428 at SERNOX:  brad | 2005-08-20 19:09:26 +0200
   r5506 at buttercup:  j0j0 | 2005-08-20 11:15:54 -0600
   Restructure torture_join_domain() so that it joins itself, removes itself, and joins itself to the domain again to ensure that its account information is all current and as expected.

  r9452 at SERNOX:  brad | 2005-08-21 19:33:51 +0200
   r5508 at buttercup:  j0j0 | 2005-08-21 11:40:36 -0600
   Bugfixes, trying to get things straight between contexts.

  r9467 at SERNOX:  brad | 2005-08-22 04:00:48 +0200
   r5510 at buttercup:  j0j0 | 2005-08-21 20:06:55 -0600
   Another round of bugfixing.

  r9521 at SERNOX:  brad | 2005-08-23 15:26:44 +0200
   r5596 at buttercup:  j0j0 | 2005-08-23 07:33:06 -0600
   Merging changes

  r9524 at SERNOX:  metze | 2005-08-23 16:09:42 +0200
  - fix the build caused by changes in the main samba4 tree,
  - add an option "dssync:german=yes" to allow me to run against my german w2k3 server
    this should be replaces by CLDAP calls to get the Default-First-Site-Name dynamicly
  - remove some temporary comments, as DsAddEntry works now

  metze
  r9528 at SERNOX:  metze | 2005-08-23 18:22:22 +0200
  the RPC-DSSYNC test is now able to fetch the whole tree,
  including the unicodePwd, ntPwdHistory fields

  metze
  r9559 at SERNOX:  brad | 2005-08-24 04:11:47 +0200
   r5612 at buttercup:  j0j0 | 2005-08-23 20:19:12 -0600
   Some fixes around using talloc in a hierarchical fashion. 
   Still not right, but better.

  r9564 at SERNOX:  brad | 2005-08-24 05:43:11 +0200
   r5614 at buttercup:  j0j0 | 2005-08-23 21:50:38 -0600
   Gave libnet_JoinADSDomain() its own tmp_ctx rather than passing it from libnet_JoinDomain() as a parameter (yuk).

   As a side effect, it proves that my bug lies in libnet_JoinDomain(), not libnet_JoinADSDomain().

  r9565 at SERNOX:  brad | 2005-08-24 06:09:46 +0200
   r5616 at buttercup:  j0j0 | 2005-08-23 22:17:12 -0600
   Small fix, if r->out.error_string and r2->samr_handle.out.error_string weren't set to NULL, torture_join_domain() would segfault on the second join.

  r9630 at SERNOX:  brad | 2005-08-26 06:42:50 +0200
  Commented out the parts of the dssync test which perform the dc join and create/remove associated ldap entries.

  Commented out the test for the 'german' dssync option, because now we detect the Site-Name using CLDAP. If cldap_netlogon() does not return ok, the code defaults to 'Default-First-Site-Name'.
  r9670 at SERNOX:  brad | 2005-08-27 02:30:11 +0200
  Added a patch from metze.

  To showcase what i've learned today, i've created two new parameters which can be set at runtime, drsuapi:last_usn and drsuapi:partition.

  drsuapi:last_usn takes an integer representing the USN of the last recieved replication update for a particular partition (uses the domain dn if drsuapi:parition isn't set). 

  That value is passed in the DsGetNCChanges() call so that only info which has been updated since that point in time is returned. If this option is not set, 0 is used by default, and all updates for that partition are returned.

  drsuapi:partition takes a string dn and uses that as the name of the AD partition to replicate.

  Some debugging output was also added. 

  r9723 at SERNOX:  brad | 2005-08-29 01:07:51 +0200
  Added some copyright notices.

  Changed some things in net_join.c to try and figure out why 'net join <domain> bdc' segfaults. 
  It occurs when the last talloc_free() happens, so i'm sure it's something to do with the memory fiddling i'm doing in libnet_join. 

  Added some drsuapi attribute ids that I figured out today. 
  I put some (many, dry) notes together while doing that, so i'll try to put them up on a blog at samba.org a little later tonight.

  r9740 at SERNOX:  metze | 2005-08-29 16:58:03 +0200
  fix up the DsGetNCchanges loop,
  and remove misleading comments

  metze
  r9743 at SERNOX:  metze | 2005-08-29 17:26:45 +0200
  make the logic a bit clearer

  metze
  r9815 at SERNOX:  brad | 2005-08-31 02:36:21 +0200
  Added cldap_netlogon() AD Site-Name lookup into libnet/libnet_join.c.
  Bugfixing rampage in libnet_join.c to resolve misunderstanding of talloc_steal().
  libnet_join now creates the CN=<netbios name>,CN=Servers,CN=<site name>,CN=Sites,CN=Configuration,<domain dn> container on a dc join.

  r9858 at SERNOX:  brad | 2005-09-01 03:17:17 +0200
  Removed extraneous NDR_ALL subsystem requirement from torture/config.mk.
  Added lots of error checking as per metze's advice.
  Removed commented out code.
  More bug chasing.

  r9863 at SERNOX:  brad | 2005-09-01 05:53:19 +0200
  Cleaned up dssync.c, removed the unneeded DsCrackNames() call, removed DC join/leave related stuff. 
  It no longer looks like my house does!

  r9887 at SERNOX:  metze | 2005-09-01 11:34:03 +0200
  - fix dssync:highest_usn parameter handling
  - ask for LINKED_ATTRIBUTE replication

  metze
  r9891 at SERNOX:  metze | 2005-09-01 14:13:18 +0200
  make the code more readable, and fix a few bugs

  metze
  r9911 at SERNOX:  brad | 2005-09-01 20:36:27 +0200
  Bugfixes in libnet_join.c.
  Cleaned up comments.
  Added domain_dn_str and account_dn_str to struct libnet_JoinDomain.
  Removed struct dcerpc_pipe *samr_pipe and struct policy_handle user_handle from struct libnet_Join.

  r9920 at SERNOX:  brad | 2005-09-01 23:34:13 +0200
  Added disclaimer (I can't seem to get libnet_JoinDomain() to keep the samr_pipe and u_handle open past the function call, grrrr....).

  r9921 at SERNOX:  brad | 2005-09-01 23:37:54 +0200
  Added copyright statement.
  Cleaned up unneeded variables from torture_join_domain().

  r9932 at SERNOX:  brad | 2005-09-02 01:49:42 +0200
  Really rushed project notes.

  r10841 at SERNOX:  metze | 2005-10-08 20:01:45 +0200
  remove diff to main SAMBA_4_0 branch

  metze
  r10862 at SERNOX:  metze | 2005-10-10 10:31:52 +0200
  remove the differences between SAMBA_4_0 and SOC/SAMBA_4_0

  metze
  r10863 at SERNOX:  metze | 2005-10-10 10:34:26 +0200
  fix the build 

  metze
  r10864 at SERNOX:  metze | 2005-10-10 11:10:08 +0200
  remove README file to reduce, diffs to main SAMBA_4_0 branch:

  metze

  README:
  This project was centered around adding a torture test to Samba 4, which used drsuapi_DsGetNCChanges() to retrieve the contents of an Active Directory in the same manner as an Active Directory DC replication event.

  As the project unfolded, I also applied some changes to the functionality of the libnet library related to joining a machine account to a domain.

  One of the first things that I implemented in this project was a 'neighbour_writeable' option for the RPC-DRSUAPI torture test. The command line to execute this torture test is as follows:

  smbtorture --option=drsuapi:neighbour_writeable=True -W <domain name> -U <admin username>%<password> ncacn_ip_tcp:<domain controller dns name> RPC-DRSUAPI

  This option provides us with runtime control over the DRSUAPI_DS_REPLICA_NEIGHBOUR_WRITEABLE flag in the struct drsuapi_DsGetNCChanges.in.req.req<level>.replica_flags, allowing us to easily test for differences in the behaviour of AD replication with the switch on or off.

  In the course of the project, I also implemented two more flags for the RPC-DSSYNC test. dssync:last_usn takes an integer representing the USN (Universal Serial Number) of the last recieved replication update for a particular partition (uses the domain DN if drsuapi:parition isn't set).   That value is passed in the DsGetNCChanges() call so that only info which has been updated since that point in time is returned. If this option is not set, 0 is used by default, and all updates for that partition are returned.  dssync:partition takes a string DN and uses that as the name of the AD partition to replicate.

  Based initially on a patch provided to me by one of my mentors, Stephan (metze) Metzmacher, the RPC-DSSYNC test was implemented for this project. Initially functionality was included to perform a DC join prior to initiating replication, but the code was removed when it was realized that replication could indeed take place without being a member of the domain in any way. It has been recently suggested that we may need a DC join after all to get all of the information we may want from the AD replication. This is probably best added using a torture_join_domain() call once the libnet code is able to keep the user policy handle and SAMR RPC pipe open.

  The DC join code was taken out of the RPC-DSSYNC and implemented for the most part in the libnet libraries. To test this, the RPC-NETLOGON test was modified to perform a domain join, leave and rejoin. Currently, the test has a fault in that it is unable to leave the domain using the same SAMR RPC pipe and user_policy information as was used for the first join. This is because I was unable to get the code working properly in libnet to provide that functionality. Currently missing from the DC join in libnet is the code to create the CN=NTDS Settings,CN=<DC NETBIOS NAME>,CN=<Site-Name>,CN=Sites,CN=Configuration,<domain DN> container using the dcerpc_drsuapi_DsAddEntry() call. I did not want to implement this functionality in libnet while there were still problems with the code.

  I also provided the ability in libnet and the RPC-DSSYNC test to look up the proper site name using the cldap library.

  In my investigations, I was unable to find out any information regarding the UnicodePwd attribute, except that the same password is represented differently for two different users in the same directory.

  I was also able to resolve and confirm the meaning of some DRSUAPI_ATTRIBUTE ID's.
  	DRSUAPI_OBJECTCLASS_domain 	(0xA0042)
  	DRSUAPI_OBJECTCLASS_domainDNS 	(0xA0043)
  	wellKnownObjects 		(0x9026A)
  	fSMORoleOwner 			(0x90171)
  	name or dc			(0x90001)
  	whenCreated 			(0x20002)
  	instanceType 			(0x20001)
  	gPLink				(0x9037B)
  These were added to the IDL for drsuapi (source/librpc/idl/drsuapi.idl).

  I would like to thank everyone on the Samba team who worked with me and assisted me with this project, specifically all the work done by Stephan Metzmacher, Andrew Bartlett and Jerry Carter. Working on this project with the Samba team really has been a life changing experience, as corny as that sounds. 

  I've realized that I was born to be a systems developer, and it has helped confirm in my mind that Open Source (specifically Samba) development is exactly what i've been missing! 

  I would also like to take this opportunity to thank Chris Dibona and Google for the amazing opportunity. I don't know if I would have taken the leap in other circumstances.

  I know these notes sound a little rushed, but it is 23:55 after all! :)

 r13574 at SERNOX (orig r10867):  metze | 2005-10-10 13:21:02 +0200
 add WERR_UNKNOWN_REVISION errorcode

 metze
 r13575 at SERNOX (orig r10868):  metze | 2005-10-10 13:47:23 +0200
 make flag(NDR_PAHEX) possible to use and show the union level in hex

 metze
 r13576 at SERNOX (orig r10869):  metze | 2005-10-10 14:10:10 +0200
 add dummy functions and dummy parsing of XPRESS decompression,
 this is the compression algorithm used by w2k3 for DsGetNCChanges().

 This algorithm isn't known yet, but it seems to be some sort of Lempel-Ziv
 algorithm.

 metze
 r13577 at SERNOX (orig r10870):  metze | 2005-10-10 14:14:29 +0200
 decompress DsGetNCChangesCtr7 replies, that uses type 2 (MSZIP)
 compression

 metze
 r13578 at SERNOX (orig r10871):  metze | 2005-10-10 14:31:05 +0200
 make xpress compression choosable, by --option="dssync:xpress=yes",
 default is to not use it, as it's currently not supported

 metze
 r13579 at SERNOX (orig r10872):  metze | 2005-10-10 15:25:11 +0200
 fix the length of the dummy XPRESS decompressed buffer

 metze
 r13580 at SERNOX (orig r10873):  metze | 2005-10-10 16:10:37 +0200
 check the complete payload header

 metze

Added:
   branches/tmp/samba4-winsrepl/source/torture/rpc/dssync.c
Modified:
   branches/tmp/samba4-winsrepl/
   branches/tmp/samba4-winsrepl/source/include/doserr.h
   branches/tmp/samba4-winsrepl/source/libcli/util/doserr.c
   branches/tmp/samba4-winsrepl/source/librpc/idl/drsuapi.idl
   branches/tmp/samba4-winsrepl/source/librpc/ndr/libndr.h
   branches/tmp/samba4-winsrepl/source/librpc/ndr/ndr_basic.c
   branches/tmp/samba4-winsrepl/source/librpc/ndr/ndr_compression.c
   branches/tmp/samba4-winsrepl/source/pidl/lib/Parse/Pidl/Samba/NDR/Parser.pm
   branches/tmp/samba4-winsrepl/source/torture/config.mk
   branches/tmp/samba4-winsrepl/source/torture/rpc/drsuapi.c
   branches/tmp/samba4-winsrepl/source/torture/torture.c

Changeset:
Sorry, the patch is too large (861 lines) to include; please use WebSVN to see it!
WebSVN: http://websvn.samba.org/cgi-bin/viewcvs.cgi?view=rev&root=samba&rev=10875