[Samba] Azure AD Connect and replication issues

[Michal Bruncko]

ups, seems pictures (attachments in general) are not accepted here,
screen (graph) is available here: 
https://i.postimg.cc/xCk6k038/image-2020-10-21-190940.png

On 10/21/2020 6:00 PM, Michal Bruncko wrote:
> hello
>
> our AD domain is hosted by two samba AD domain controllers version 4.12.6
> - replication between controllers is fine, no problems.
> - no schema errors.
> - no database errors, all fine.
> - no CPU utilizations
> - wthout noticeable bandwidth utilization
>
> Recently we have deployed Azure AD connector on dedicated windows 
> system (system is domain member server). since this deployment we are 
> observing following issues on DCs:
> - CPU utilization issue (one CPU core fully utilized)
> - high BW utilization
> - replication issue messages:
> [2020/10/21 17:41:55.043563,  0] 
> ../../source4/rpc_server/drsuapi/getncchanges.c:2910(dcesrv_drsuapi_DsGetNCChanges)
>   ../../source4/rpc_server/drsuapi/getncchanges.c:2910: DsGetNCChanges 
> 2nd replication on DN DC= older highwatermark (last_dn 
> CN=userXYZ,OU=Users,DC=)
>
>
> and this is happening only on one DC server in time - the one, to 
> which this AD connector is connected for doing AD to AAD sync tasks.
>
> More details:
>
> CPU: mostly only one CPU core from all system-assigned cores is 
> utilized at 100%:
>
>
> BW utilization: you can see example here (peak starts once the Azure 
> AD connector connects to particular DC server) (notice the "uploaded" 
> data - 54GB - value from DC system):
>
>
>
> Replicaton errors: repeating messages (example above) every each 4-5 
> seconds. the "last_dn" is changing during time slowly: it is changed 
> to another (user) object each several hours.
>
> no other issues observed.
>
> - If we deactivate this Azure connector, all issues stopped (but of 
> course we are out of sync with AAD)
> - if we reboot/stop DC1 services (serving for Azure connector), the 
> Azure connector switch to DC2 and same story happen again 
> (CPU/bandwidth/replication logs)
>
> I've found similar issue reported back in 2017: 
> https://lists.samba.org/archive/samba/2017-October/211756.html 
> ([Samba] samba getting stuck, highwatermark replication issue?)
>
> seems this issue is still in place now. no difference.
>
>
> does anyone else have similar issues? does anyone else how to resolve 
> them? either on Azure AD connector side (there are various 
> confiuration option available) or (possibly) on samba side?
>
>
> thank you
> michal
>