[Samba] AD DC out of sync

[Dr. Lars Hanke]

It did happen again and this time I was a little less panicked and took 
some time to figure out what happened.

On my primary DC (SAMBA) I did not notice anything extraordinary. 
However, my secondary (VERDANDI) reported issues:

root at verdandi:~# samba-tool drs showrepl
Default-First-Site-Name\VERDANDI
DSA Options: 0x00000001
DSA object GUID: a03bbb51-1dca-44ae-a4d9-7aa8cb4a1ace
DSA invocationId: 8bdb4f85-1da2-4f5a-b9a9-e8369d202745

==== INBOUND NEIGHBORS ====

CN=Schema,CN=Configuration,DC=ad,DC=microsult,DC=de
         Default-First-Site-Name\SAMBA via RPC
                 DSA object GUID: b19509be-c3ee-4a58-9fc9-afd61759a23f
                 Last attempt @ Wed Apr 22 00:12:36 2015 CEST failed, 
result 5 (WERR_ACCESS_DENIED)
                 1265 consecutive failure(s).
                 Last success @ Fri Apr 17 14:47:18 2015 CEST

[...]
==== OUTBOUND NEIGHBORS ====
[... everything OK for no attempts were ever made, but ...]

DC=ad,DC=microsult,DC=de
         Default-First-Site-Name\SAMBA via RPC
                 DSA object GUID: b19509be-c3ee-4a58-9fc9-afd61759a23f
                 Last attempt @ Wed Apr 22 00:14:00 2015 CEST failed, 
result 5 (WERR_ACCESS_DENIED)
                 31 consecutive failure(s).
                 Last success @ NTTIME(0)

And consequently the password update that happened the previous day was 
out of sync:

samba-tool ldapcmp ldap://samba ldap://verdandi -Uadministrator
Password for [AD\administrator]:

* Comparing [DOMAIN] context...

* Objects to be compared: 289

Comparing:
'CN=Builtin,DC=ad,DC=microsult,DC=de' [ldap://samba]
'CN=Builtin,DC=ad,DC=microsult,DC=de' [ldap://verdandi]
     Attributes found only in ldap://samba:
         serverState
     FAILED

Comparing:
'CN=Lars LH. Hanke,CN=Users,DC=ad,DC=microsult,DC=de' [ldap://samba]
'CN=Lars LH. Hanke,CN=Users,DC=ad,DC=microsult,DC=de' [ldap://verdandi]
     Difference in attribute values:
         pwdLastSet =>
['130740170160000000']
['130703672860000000']
     FAILED

[...]

Having restarted the secondary DC some 34h ago, it synchronized 
immediately and still does, i.e. drs showrepl has its last success 5 
minutes ago, no failures.

It looks a little like an expired ticket, which fails to renew after 
several weeks. But this is pure speculation.

Any ideas for troubleshooting?

Regards,
  - lars.

Am 13.03.2015 um 00:43 schrieb Lars Hanke:
> Hi Marc,
>
>  >> The cause is that the password change didn' reach both AD DCs, but only
>>> one. The other one still had the old value as could be seen by
>>> samba-tool ldapcmp. Restarting the DCs and waiting for a couple of
>>> seconds brings them back to sync and Windows logons work as they used
>>> to.
>>> Any idea, what I should do next time to obtain valuable output for
>>> debugging?
>>
>> * What Samba version are you running?
>
> The DCs are 4.1.17-Debian.
>
>> * How many DCs?
>
> Just two.
>
>> * Can you force this problem to appear?
>
> Need some more investigation here - I did not find any way reproducible
> under arbitrary conditions.
>
>> Just an idea: AD problems are often caused by DNS problems and we got
>> the keyword "DNS islanding" in an other threat at the moment: Which DNS
>> do your DCs use as primary? Their own or a different one? See
>> http://retrohack.com/a-word-or-two-about-dns-islanding/
>
> As I understood Linux resolving there is no static primary-secondary
> concept for DNS. So I'll try to remove the self-dependence altogether
> and see, if it enhances the situation.
>
> Regards,
>   - lars.
>