[Samba] Problems with Samba 4.5.16 - configuring a second failover AD DC and joining this to an existing domain SAMDOM

[Stephen]

Thanks for taking a look Roland, and I appreciate your comments 
regarding your scripts. I am not a professional sysadmin so there likely 
is stuff there that the grizzled unix veterans on this list will find a 
little odd  :)

I just restarted samba on ad2 as per your suggestion and I got the 
following output:

pi at ad2:~ $ sudo systemctl restart samba-ad-dc.service
pi at ad2:~ $ sudo samba-tool drs showrepl
Default-First-Site-Name\AD2
DSA Options: 0x00000001
DSA object GUID: e676dfc3-670d-46bb-b1f7-756bae990a30
DSA invocationId: b7fb9a73-a5c5-4672-9d0f-83e0323f9f3b

==== INBOUND NEIGHBORS ====

CN=Configuration,DC=samdom,DC=example,DC=com
         Default-First-Site-Name\AD1 via RPC
                 DSA object GUID: a021ecef-e1f1-41ea-9787-9c3678f25e4a
                 Last attempt @ Fri Mar 22 16:16:01 2019 GMT was successful
                 0 consecutive failure(s).
                 Last success @ Fri Mar 22 16:16:01 2019 GMT

DC=DomainDnsZones,DC=samdom,DC=example,DC=com
         Default-First-Site-Name\AD1 via RPC
                 DSA object GUID: a021ecef-e1f1-41ea-9787-9c3678f25e4a
                 Last attempt @ Fri Mar 22 16:16:00 2019 GMT was successful
                 0 consecutive failure(s).
                 Last success @ Fri Mar 22 16:16:00 2019 GMT

CN=Schema,CN=Configuration,DC=samdom,DC=example,DC=com
         Default-First-Site-Name\AD1 via RPC
                 DSA object GUID: a021ecef-e1f1-41ea-9787-9c3678f25e4a
                 Last attempt @ Fri Mar 22 16:16:01 2019 GMT was successful
                 0 consecutive failure(s).
                 Last success @ Fri Mar 22 16:16:01 2019 GMT

DC=ForestDnsZones,DC=samdom,DC=example,DC=com
         Default-First-Site-Name\AD1 via RPC
                 DSA object GUID: a021ecef-e1f1-41ea-9787-9c3678f25e4a
                 Last attempt @ Fri Mar 22 16:16:00 2019 GMT was successful
                 0 consecutive failure(s).
                 Last success @ Fri Mar 22 16:16:00 2019 GMT

DC=samdom,DC=example,DC=com
         Default-First-Site-Name\AD1 via RPC
                 DSA object GUID: a021ecef-e1f1-41ea-9787-9c3678f25e4a
                 Last attempt @ Fri Mar 22 16:16:01 2019 GMT was successful
                 0 consecutive failure(s).
                 Last success @ Fri Mar 22 16:16:01 2019 GMT

==== OUTBOUND NEIGHBORS ====

==== KCC CONNECTION OBJECTS ====

Connection --
         Connection name: c2f83b11-0d06-41d3-b6c1-438ae935852c
         Enabled        : TRUE
         Server DNS name : ad1.samdom.example.com
         Server DN name  : CN=NTDS 
Settings,CN=AD1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=samdom,DC=example,DC=com
                 TransportType: RPC
                 options: 0x00000001
Warning: No NC replicated for Connection!

The LDAP error doesn't appear to reoccur after restart, but should the 
outbound neighbours section be empty here in this output? That seems odd.
To further verify correctness and overall sanity I tried to create some 
user accounts, both on ad1 and ad2. I would expect these accounts to 
automatically replicate across all DCs if everything is working properly.

First on ad1:
pi at ad1:~ $ sudo samba-tool user add test_user
Note: samba-tool user add is deprecated.  Please use samba-tool user 
create for the same function.
New Password:
Retype Password:
User 'test_user' created successfully

Verifying this on ad2:
pi at ad2:~ $ sudo samba-tool user list
Administrator
test_user
krbtgt
Guest

[WORKS OK]

Likewise on ad2:
pi at ad2:~ $ sudo samba-tool user add test_user2
Note: samba-tool user add is deprecated.  Please use samba-tool user 
create for the same function.
New Password:
Retype Password:
User 'test_user2' created successfully

Verifying on ad1:

pi at ad1:~ $ sudo samba-tool user list
Administrator
test_user
krbtgt
Guest

[FAILURE, test_user2 doesnt replicate as expected]

So something is clearly wrong here!

  * I can create users on ad1 and these transfer to ad2 without issue

  * However the converse is not true, users created on ad2 do not seem
    to transfer to ad1.

I did try a sudo systemctl restart samba-ad-dc after each user creation, 
but that didn't seem to make a difference.

Thanks
Stephen