[Samba] DsReplicaSync WERR_LOGON_FAILURE in one direction only (fresh DC, Kerberos auth verified working) - Samba 4.22.10 on Debian 13

[Johann Stieger]

Hi, everyone,

Replication between two Samba AD DCs works reliably in one direction (DC1 -> DC2) but consistently fails in the other direction (DC2 -> DC1) with WERR_LOGON_FAILURE  (1326), even immediately after a completely fresh samba-tool domain join . This has now been reproduced identically across two separate demote+rejoin cycles, a week apart, with the same result each time.

Unusually, I've been able to rule out a genuine Kerberos/credential problem: manual kinit -k  with the freshly-provisioned keytab succeeds, and kvno  against the exact DRSUAPI SPN on the target DC also succeeds - yet the DRS bind itself still fails with LOGON_FAILURE.

Environment

OS: Debian 13 (Trixie), both DCs

Samba version: 4.22.10-Debian-4.22.10+dfsg-0+deb13u2 (both DCs)

DC1 (existing, stable, uptime several days at time of test): rpi5b, 192.168.222.16

DC2 (freshly joined): rpi5d, 192.168.222.18

DNS backend: SAMBA_INTERNAL

Domain/forest functional level: 2008 R2

Both DCs on the same subnet, no firewall between them (verified via nc on ports 88/389/445 in both directions)

Time sync verified to sub-second accuracy between both hosts (chrony, DC2 syncs against DC1)

Steps to reproduce

On DC1 (rpi5b), demote and remove the previous DC2 entry cleanly:

 samba-tool domain demote --remove-other-dead-server=RPI5D -U administrator

On DC2 (rpi5d), wipe local state and rejoin fresh:

 rm -rf /var/lib/samba (moved aside) rm /etc/samba/smb.conf (moved aside) mkdir -p /var/lib/samba samba-tool domain join ad.domain.tld DC -U administrator --dns-backend=SAMBA_INTERNAL

Immediately after join (before touching smb.conf or restarting anything else), verify the freshly-provisioned keytab works:

 kinit -k 'RPI5D$@AD.DOMAIN.TLD' -t /var/lib/samba/private/secrets.keytab klist # succeeds, valid TGT issued

Start samba-ad-dc, wait 20+ minutes without touching anything else.

Check replication status from both sides.

Observed result

>From DC1's perspective ( samba-tool drs showrepl rpi5d.ad.domain.tld -U administrator , run on DC1):

==== INBOUND NEIGHBORS ==== DC=ad,DC=domain,DC=tld Default-First-Site-Name\RPI5B via RPC Last attempt @ ... was successful 0 consecutive failure(s). [all 5 partitions: same, all successful] ==== OUTBOUND NEIGHBORS ==== (empty)

>From DC2's perspective ( samba-tool drs showrepl rpi5b.ad.domain.tld -U administrator , run on DC2), taken at the same time:

==== INBOUND NEIGHBORS ==== DC=ad,DC=domain,DC=tld Default-First-Site-Name\RPI5D via RPC Last attempt @ ... failed, result 1326 (WERR_LOGON_FAILURE) 4 consecutive failure(s). Last success @ NTTIME(0) [all 5 partitions: same failure]

So DC1 -> DC2 replication succeeds automatically and repeatedly. DC2 -> DC1 never succeeds, not even once, from the moment of join onward.

What I've ruled out

Network/firewall : nc -zv  succeeds on ports 445/389/88 in both directions; ping  succeeds in both directions.

DNS : SRV records for both DCs present and correct on both sides; host -t A  resolves correctly for both hostnames from both hosts.

/etc/hosts interference : found and fixed a Debian default 127.0.1.1 <hostname>.<domain> <hostname>  entry that was causing a different , earlier symptom ( NT_STATUS_CONNECTION_REFUSED from 127.0.1.1  during the GSSAPI phase) - after fixing that (keeping only the short hostname on 127.0.1.1, letting the FQDN resolve via Samba's internal DNS), the GSSAPI handshake completes cleanly, but the LOGON_FAILURE on DsReplicaSync itself persists.

Time skew : confirmed sub-second sync between both hosts via chrony at the moment of failure.

Kerberos ticket/keytab validity : kinit -k  with the exact keytab Samba just provisioned succeeds on the DC2 side. kvno  for both HOST/rpi5b.ad.domain.tld at REALM  and the specific DRSUAPI SPN ( E3514235-4B06-11D1-AB04-00C04FC2DCD2/<DC1-GUID>/<realm>@REALM ) succeeds from DC2 against DC1.

msDS-KeyVersionNumber : identical between what's in DC1's copy of sam.ldb for the RPI5D computer object and what's in DC2's local keytab (both showed KVNO 2 at time of comparison).

msDS-SupportedEncryptionTypes : identical (28) on both computer objects.

Group membership : both computer accounts correctly listed under "Domain Controllers" group.

SPN completeness : samba-tool ldapcmp  between the two DCs showed DC1's copy of the DC2 computer object had fewer SPNs than DC2's own local copy (the additional SPNs - ldap/DC2, RestrictedKrbHost/DC2, GC/DomainDnsZones/ForestDnsZones variants - apparently never made it back to DC1 due to the one-way replication break itself, a chicken-and-egg problem). Manually adding the missing SPNs via samba-tool spn add  on DC1 did not  resolve the issue.

Stale in-memory state on DC1 : restarting samba-ad-dc on DC1 (which had an uptime of several days spanning multiple previous demote/rejoin cycles) did not  resolve the issue.

-k yes  flag : deprecated in this version, no effect either way.

Debug output at point of failure (debuglevel=10, run on DC2)

Starting GENSEC mechanism spnego Starting GENSEC submechanism gssapi_krb5 GSSAPI credentials for RPI5D$@AD.DOMAIN.TLD will expire in 36000 secs gensec_gssapi: NO credentials were delegated GSSAPI Connection will be cryptographically sealed [... successful sealed rpc request/response exchange follows ...] drsuapi_DsReplicaSync: struct drsuapi_DsReplicaSync out: struct drsuapi_DsReplicaSync result : WERR_LOGON_FAILURE

Notably, this shows the GSSAPI/Kerberos handshake completing successfully (sealed connection established, request sent, response received) - the LOGON_FAILURE is returned as the application-level result  of the DsReplicaSync call itself, not as an RPC bind/auth failure. This is different from the more commonly-reported case where the bind itself fails at the GENSEC layer.

On DC1's side, the continuously-recurring journal messages during the DC1->DC2 direction (the one that does  work) show the expected:

Failed to bind to uuid e3514235-4b06-11d1-ab04-00c04fc2dcd2 for ncacn_ip_tcp:192.168.222.18[49153,seal,krb5,...] NT_STATUS_LOGON_FAILURE

This appears to be periodic retry noise from the notify-based push replication for the direction that is broken (DC2 trying to notify/pull against DC1), not related to the successful direction.

Question

Has anyone seen WERR_LOGON_FAILURE  returned specifically as the result  of drsuapi_DsReplicaSync  (as opposed to a GENSEC/RPC bind failure) when the underlying Kerberos service ticket for the DRS SPN is independently verified to be obtainable? Is there a known Samba 4.22.x regression around this, or a specific ACL/attribute on the domain object (beyond standard "Domain Controllers" membership) that governs this at the DRSUAPI application layer rather than the RPC/auth layer?

Happy to run any additional diagnostics if pointed in a direction - this has been reproduced identically twice, a week apart, with a clean demote+rejoin between attempts, so I'm fairly confident it's not transient state corruption.

Thanks in advance and best regards,

Johann