[Samba] Samba 4.3.x high CPU load

Wed Jan 6 10:07:08 UTC 2016

On 06/01/16 09:08, Chris Alavoine wrote:
> Hi there,
>
> I have a multi DC global setup. 9 x Ubuntu 14.04.3 DC's in multiple Sites.
>
> This has been working nicely for some time however recently the FSMO holder
> has been refusing LDAP requests on occasions and showing constant very high
> CPU usage:
>
> top - 08:59:12 up  8:51,  1 user,  load average: 1.03, 1.00, 1.03
> Tasks: 186 total,   4 running, 182 sleeping,   0 stopped,   0 zombie
> %Cpu0  :  2.6 us,  2.6 sy,  0.0 ni, 94.9 id,  0.0 wa,  0.0 hi,  0.0 si,
>   0.0 st
> %Cpu1  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,
>   0.0 st
> %Cpu2  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,
>   0.0 st
> %Cpu3  : 97.4 us,  2.6 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,
>   0.0 st
> %Cpu4  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,
>   0.0 st
> %Cpu5  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,
>   0.0 st
> %Cpu6  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,
>   0.0 st
> %Cpu7  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,
>   0.0 st
> KiB Mem:   4078212 total,  2193268 used,  1884944 free,   354864 buffers
> KiB Swap:  1949692 total,        0 used,  1949692 free.  1010792 cached Mem
>
>    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
> 25571 root      20   0  839960 288416  30328 R  99.5  7.1  56:04.45 samba
>    968 bind      20   0 1097008  89808   8168 S   2.6  2.2   6:57.09 named
>
>
> I am also seeing this if I do "samba-tool fsmo show":
>
> ERROR(<type 'exceptions.KeyError'>): uncaught exception - 'No such element'
>    File
> "/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/__init__.py",
> line 175, in _run
>      return self.run(*args, **kwargs)
>    File "/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/fsmo.py",
> line 395, in run
>      domaindnszonesMaster = get_fsmo_roleowner(samdb, domaindns_dn)
>    File "/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/fsmo.py",
> line 42, in get_fsmo_roleowner
>      master_owner = res[0]["fSMORoleOwner"][0]
>
> If I stop/start samba the high load switches to the other DC in this Site
> and the same behaviour is exhibited.
>
> Has anyone else experience anything like this? Could it be linked to the
> recent patch for CVE-2015-5330 (Remote memory read in Samba LDAP server)?
> I've tried patching my main FSMO roles DC and it's Site counterpart. My
> other DC's are still on 4.3.1, but I am planning to upgrade them today. The
> high load still persists on the 4.3.3 upgraded DC's, so I'm guessing this
> is something else.
>
> We use NSLCD bindpw to authenticate the majority of our member servers.
> This has worked very well for a few years now but could there be a problem
> there maybe? This is our nslcd conf:
>
> uid nslcd
> gid nslcd
> uri ldap://192.168.x.x ldap://192.168.x.x
> base dc=EXAMPLE,dc=internal,dc=com
> binddn CN=ldap-connect,CN=Users,DC=example,DC=internal,DC=com
> bindpw xxxxxxxxxxxxxx
> pagesize 1000
> referrals off
> filter  passwd  (objectClass=user)
> filter  group   (objectClass=group)
> map     passwd   uid                sAMAccountName
> map     passwd  homeDirectory      unixHomeDirectory
>
>
> Any pointers much appreciated.
>
> Thanks,
> Chris.
>

I think this is your problem:

ERROR(<type 'exceptions.KeyError'>): uncaught exception - 'No such element'

When I run 'samba-tool fsmo show' I get:

SchemaMasterRole owner: CN=NTDS 
Settings,CN=DC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=samdom,DC=example,DC=com
InfrastructureMasterRole owner: CN=NTDS 
Settings,CN=DC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=samdom,DC=example,DC=com
RidAllocationMasterRole owner: CN=NTDS 
Settings,CN=DC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=samdom,DC=example,DC=com
PdcEmulationMasterRole owner: CN=NTDS 
Settings,CN=DC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=samdom,DC=example,DC=com
DomainNamingMasterRole owner: CN=NTDS 
Settings,CN=DC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=samdom,DC=example,DC=com
DomainDnsZonesMasterRole owner: CN=NTDS 
Settings,CN=DC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=samdom,DC=example,DC=com
ForestDnsZonesMasterRole owner: CN=NTDS 
Settings,CN=DC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=samdom,DC=example,DC=com

Try this:

ldbsearch --cross-ncs -H /usr/local/samba/private/sam.ldb 
'(fsmoroleowner=*)' | grep 'dn:' | sed 's|dn: ||'

it should return something like this:

CN=Schema,CN=Configuration,DC=samdom,DC=example,DC=com
CN=Partitions,CN=Configuration,DC=samdom,DC=example,DC=com
CN=Infrastructure,DC=DomainDnsZones,DC=samdom,DC=example,DC=com
CN=Infrastructure,DC=ForestDnsZones,DC=samdom,DC=example,DC=com
CN=Infrastructure,DC=samdom,DC=example,DC=com
DC=samdom,DC=example,DC=com
CN=RID Manager$,CN=System,DC=samdom,DC=example,DC=com

You can find out who owns the individual fsmorole with:

ldbsearch --cross-ncs --show-binary -H /usr/local/samba/private/sam.ldb 
-b 'CN=Infrastructure,DC=DomainDnsZones,DC=samdom,DC=example,DC=com' 
'(fsmoroleowner=*)' | grep fSMORoleOwner | sed 's|fSMORoleOwner: ||'

This should return something like this:

CN=NTDS 
Settings,CN=DC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=samdom,DC=example,DC=com

I get the feeling it will not return anything for the 
domaindnszonesMaster role (and possible also the forestdnszonesmaster role)

Rowland