[Samba] Azure AD Connect and replication issues / WORKAROUND

Sun Oct 25 16:11:03 UTC 2020

hello all

just to summarize this topic for anybody else.
after some time spent with tshooting there seems is a sufficient 
workaround for this issue: disable "sync of password hashes" option 
within Azure AD connect tool.
with disabling this option all three issues (CPU/Bandwidth 
utilization/replication logs) are gone!
seems that for now the only working option for samba and Azure connect 
tool is to:
- use pass-through authentication
- deploy at least one authentication agent (agent is installed a part of 
Azure AD connect tool), but two agents are recommended
- and disable "sync of password hashes" option in Azure AD connect tool

seems that syncing password hahes is still problem for samba: 
https://bugzilla.samba.org/show_bug.cgi?id=10635
and it looks like Azure connect tool keeps trying (and endlessly 
retrying) to get them synced which is causing all three issues in my 
environment (CPU/Bandwidth utilization/replication logs).

Michal

On 10/22/2020 5:40 PM, Michal Bruncko via samba wrote:
> just small update:
>
> - idfix tool (Directory Synchronization Error Remediation Tool / 
> https://github.com/microsoft/idfix) shows just small issues like 
> empty/missing displayName attrib in some of objects which I have 
> corrected and no more issues present at all.
> - no errors from AAD connect event viewer: final log message is 
> "Scheduler::SchedulerThreadMain : Completed configured scheduler 
> operations." (no errors before)
> - aad connect info:
>
> Azure AD Tenant - Settings
>
> PasswordHashSync: True
> ForcePasswordChangeOnLogOn: False
> UserWriteback: False
> DeviceWriteback: False
> UnifiedGroupWriteback: False
>
>
> Azure AD Connect - Global Settings
>
> Microsoft.AADFilter.ApplicationList:
> Microsoft.AADFilter.AttributeExclusionList:
> Microsoft.Configuration.ImportDate: 2020-10-19 17:08:19Z
> Microsoft.ConnectDirectories.WizardDirectoryMode: AD
> Microsoft.DeviceWriteBack.Container:
> Microsoft.DeviceWriteBack.Forest:
> Microsoft.DirectoryExtension.SourceTargetAttributesMap:
> Microsoft.OptionalFeature.AutoUpgradeState: Enabled
> Microsoft.OptionalFeature.AutoUpgradeSuspensionReason:
> Microsoft.OptionalFeature.DeviceWriteBack: False
> Microsoft.OptionalFeature.DeviceWriteUp: True
> Microsoft.OptionalFeature.DirectoryExtension: True
> Microsoft.OptionalFeature.DirectoryExtensionAttributes:
> Microsoft.OptionalFeature.ExchangeMailPublicFolder: False
> Microsoft.OptionalFeature.ExportDeletionThreshold: True
> Microsoft.OptionalFeature.ExportDeletionThresholdValue: 500
> Microsoft.OptionalFeature.FilterAAD: True
> Microsoft.OptionalFeature.GroupFiltering: False
> Microsoft.OptionalFeature.GroupWriteBack: False
> Microsoft.OptionalFeature.HybridExchange: False
> Microsoft.OptionalFeature.UserWriteBack: False
> Microsoft.SynchronizationOption.AnchorAttribute: mS-DS-ConsistencyGuid
> Microsoft.SynchronizationOption.CustomAttribute:
> Microsoft.SynchronizationOption.JoinCriteria: AlwaysProvision
> Microsoft.SynchronizationOption.UPNAttribute: userPrincipalName
> Microsoft.Synchronize.MaintenanceEnabled: True
> Microsoft.Synchronize.NextStartTime: Thu, 22 Oct 2020 15:37:03 GMT
> Microsoft.Synchronize.RunHistoryPurgeInterval: 7.00:00:00
> Microsoft.Synchronize.ServerConfigurationVersion: 1.5.45.0
> Microsoft.Synchronize.SchedulerSuspended: False
> Microsoft.Synchronize.StagingMode: False
> Microsoft.Synchronize.SynchronizationPolicy: Delta
> Microsoft.Synchronize.SynchronizationSchedule: True
> Microsoft.Synchronize.TimeInterval: 00:30:00
> Microsoft.SystemInformation.MachineRole: RoleMemberServer
> Microsoft.UserSignIn.DesktopSsoEnabled: True
> Microsoft.UserSignIn.SignOnMethod: PassThroughAuthentication
> Microsoft.UserWriteBack.Container:
> Microsoft.UserWriteBack.Forest:
> Microsoft.Version.SynchronizationRuleImmutableTag: V1
>
>
> Azure AD Connect Sync Scheduler - Settings
>
>
> AllowedSyncCycleInterval: 00:30:00
> CurrentlyEffectiveSyncCycleInterval: 00:30:00
> CustomizedSyncCycleInterval:
> NextSyncCyclePolicyType: Delta
> NextSyncCycleStartTimeInUTC: 22.10.2020 15:37:03
> PurgeRunHistoryInterval: 7.00:00:00
> SyncCycleEnabled: True
> MaintenanceEnabled: True
> StagingModeEnabled: False
> SchedulerSuspended: False
> SyncCycleInProgress: False
>
>
>
> On 10/22/2020 1:19 PM, Michal Bruncko via samba wrote:
>> hello mj
>>
>> we are school - we are syncing approx 500 users and several groups.
>>
>> we use pass-through authentication because AAD authentication was not 
>> working (that time we used samba 4.8.12, but since moving to 4.12 I 
>> didnt tested this), we have (as recommended) one AAD connector plus 
>> two authentication agents deployed in our environment.
>>
>> the thing what I dont understand is the amount of data which is 
>> connector getting each day (~50GB). I have to check AAD connector 
>> tshoot tool to understand whether there are some issues (like sync 
>> will not get properly completed and therefore it is always retrying).
>>
>> but from AAD point of view all looks fine - in AAD portal I see the 
>> sync is done every 30 minutes without errors.
>>
>> but this is definitely not correct behaviour regards to CPU and BW 
>> utilization...
>>
>> regards
>> michal
>>
>>
>> On 10/21/2020 9:16 PM, mj via samba wrote:
>>> Hi Michal,
>>>
>>> Seems we are doing similar things at the moment: getting samba to 
>>> work with azure AD.
>>>
>>> We also see the high CPU usage on the DC that the Azure AD Connect 
>>> server connected to. Between 70 - 100 percent in our case.
>>>
>>> We are not seeing any replication issues after azure AD Connect, and 
>>> I have a script that automatically checks replication every few 
>>> minutes.
>>>
>>> I was the one reporting the highwatermark back in 2017, but we're 
>>> not getting those now. Between 2017 and now, we stopped testing 
>>> azure, and I took it up again only last week.
>>>
>>> On the samba DC side, we observe that samba is very buzy talking to 
>>> the Azure AD Connect machine, even though we filter sync using one 
>>> group containing only three members, and none of them are actually 
>>> changing. (this is just for testing now) How many users are you 
>>> syncing?
>>>
>>> Are you using pass-through authentication, or syncing password 
>>> hashes? And is what you chose working for you, authentication-wise?
>>>
>>> Regards,
>>> MJ
>>>
>>> On 10/21/20 7:10 PM, Michal Bruncko via samba wrote:
>>>> ups, seems pictures (attachments in general) are not accepted here,
>>>> screen (graph) is available here: 
>>>> https://i.postimg.cc/xCk6k038/image-2020-10-21-190940.png
>>>>
>>>> On 10/21/2020 6:00 PM, Michal Bruncko wrote:
>>>>> hello
>>>>>
>>>>> our AD domain is hosted by two samba AD domain controllers version 
>>>>> 4.12.6
>>>>> - replication between controllers is fine, no problems.
>>>>> - no schema errors.
>>>>> - no database errors, all fine.
>>>>> - no CPU utilizations
>>>>> - wthout noticeable bandwidth utilization
>>>>>
>>>>> Recently we have deployed Azure AD connector on dedicated windows 
>>>>> system (system is domain member server). since this deployment we 
>>>>> are observing following issues on DCs:
>>>>> - CPU utilization issue (one CPU core fully utilized)
>>>>> - high BW utilization
>>>>> - replication issue messages:
>>>>> [2020/10/21 17:41:55.043563,  0] 
>>>>> ../../source4/rpc_server/drsuapi/getncchanges.c:2910(dcesrv_drsuapi_DsGetNCChanges) 
>>>>>
>>>>>   ../../source4/rpc_server/drsuapi/getncchanges.c:2910: 
>>>>> DsGetNCChanges 2nd replication on DN DC= older highwatermark 
>>>>> (last_dn CN=userXYZ,OU=Users,DC=)
>>>>>
>>>>>
>>>>> and this is happening only on one DC server in time - the one, to 
>>>>> which this AD connector is connected for doing AD to AAD sync tasks.
>>>>>
>>>>> More details:
>>>>>
>>>>> CPU: mostly only one CPU core from all system-assigned cores is 
>>>>> utilized at 100%:
>>>>>
>>>>>
>>>>> BW utilization: you can see example here (peak starts once the 
>>>>> Azure AD connector connects to particular DC server) (notice the 
>>>>> "uploaded" data - 54GB - value from DC system):
>>>>>
>>>>>
>>>>>
>>>>> Replicaton errors: repeating messages (example above) every each 
>>>>> 4-5 seconds. the "last_dn" is changing during time slowly: it is 
>>>>> changed to another (user) object each several hours.
>>>>>
>>>>> no other issues observed.
>>>>>
>>>>> - If we deactivate this Azure connector, all issues stopped (but 
>>>>> of course we are out of sync with AAD)
>>>>> - if we reboot/stop DC1 services (serving for Azure connector), 
>>>>> the Azure connector switch to DC2 and same story happen again 
>>>>> (CPU/bandwidth/replication logs)
>>>>>
>>>>> I've found similar issue reported back in 2017: 
>>>>> https://lists.samba.org/archive/samba/2017-October/211756.html 
>>>>> ([Samba] samba getting stuck, highwatermark replication issue?)
>>>>>
>>>>> seems this issue is still in place now. no difference.
>>>>>
>>>>>
>>>>> does anyone else have similar issues? does anyone else how to 
>>>>> resolve them? either on Azure AD connector side (there are various 
>>>>> confiuration option available) or (possibly) on samba side?
>>>>>
>>>>>
>>>>> thank you
>>>>> michal
>>>>>
>>>>
>>>
>>
>>
>
>