[Samba] Samba 4.9.4 - high RAM usage - OOM killer

[Denis Cardon]

Hi Laurent,

>
> We upgraded a legacy (NT4) domain from 3.6 series to 4.8 and then 4.9.4
> samba version (using sernet subscription packages / debian stable)
>
> The setup is composed of 4 DCs with each 2 CPU/16GB RAM.
>
> We currently have ~700 user accounts / ~600 computers / ~150 groups
>
> Our mail setup, SSO, ... query the 4 DCs constantly.
>
>
> Every 5 to 10 days the RAM consumption and CPU usage (due to kswapd) are
> peaking.
>
> This leads to OOM killer killing samba processes
>
> kernel: [765104.826327] samba invoked oom-killer:
> gfp_mask=0x24201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0,
> order=0, oom_score_adj=0
> kernel: [765104.826355]  [<ffffffff8c3871ba>] ?
> oom_kill_process+0x21a/0x3e0
> kernel: [765104.826357]  [<ffffffff8c386e3d>] ? oom_badness+0xed/0x170
> kernel: [765104.826455] [ pid ]   uid  tgid total_vm      rss nr_ptes
> nr_pmds swapents oom_score_adj name

could you please take a look at which process is taking so much RAM? Is 
there a pile of smbd process around? If you are using Bind-dlz, how much 
memory named is eating? Is is the increase in RAM consumption is linear 
over time?

If it is a "samba" process that is eating all the RAM, please check the 
PID and the samba-tool processes command to report which part of samba 
is weaking havoc the system.

I'd say you should be fine with 4GiB with such a setup...

Cheers,

Denis


>
> ...
>
> kernel: [861216.518771] Out of memory: Kill process 603 (samba) score 3
> or sacrifice child
> kernel: [861356.048484]  [<ffffffff8c387651>] ? out_of_memory+0x111/0x470
>
> samba[614]:   ../source4/dsdb/kcc/kcc_periodic.c:768: Failed samba_kcc -
> NT_STATUS_NO_MEMORY
>
> Once this happens, the affected DC is unresponsive for all samba
> authentication processes (including LDAP).
>
> A reboot of the affected VM 'cures' the issue, but only for a short
> amount of time (5 to 10 days).
>
> Apart from either restarting samba processes on a daily basis, or
> rebooting the DCs, is there a way to:
>
> - pinpoint the root cause of the memory consumption (leak, corrupted DB,
> ...)
>
> - have the DCs use a more 'normal' amount of RAM ?
>
>
> Thanks
>
>
>
> Please note:
>
> # samba-tool drs kcc
>
> # samba-tool dbcheck --cross-ncs
>
> are not showing any errors
>
>
> :/etc/nsswitch.conf:
>
> passwd:         compat
> group:          compat
> shadow:         compat
> gshadow:        files
>
> hosts:          files dns
> networks:       files
>
> protocols:      db files
> services:       db files
> ethers:         db files
> rpc:            db files
>
>
> :/etc/samba/smb.conf:
>
> [global]
>         netbios name = VS-DC-001
>         realm = CORP.MYDOMAIN
>         workgroup = SAMBA
>
>         log file = /var/log/samba/samba.log.%m
>         log level = 1 auth_audit:3 auth_json_audit:3
>         max log size = 50000
>         debug timestamp = yes
>         dns forwarder = 192.168.0.20 192.168.100.20 192.168.0.21
> 192.168.100.21
>         server role check:inhibit=yes
>         ldap server require strong auth = no
>         wins support = yes
>         server role = active directory domain controller
>         check password script = /usr/local/bin/crackcheck -c -d
> /var/cache/cracklib/cracklib_dict
>         idmap_ldb:use rfc2307 = yes
>         server schannel = auto
>
> [netlogon]
>         path = /var/lib/samba/sysvol/corp.lncsa.com/scripts
>         read only = No
>
> [sysvol]
>         path = /var/lib/samba/sysvol
>         read only = No
>
> # du -shxc sam.ldb*
> 4.1M    sam.ldb
> 132M    sam.ldb.d
> 136M    total
>
> Samba packages:
>
> ii  samba                            99:4.9.4-10  amd64 Glue package for
> sernet-samba.
> ii  samba-common                     99:4.9.4-10  all Glue package for
> sernet-samba-common.
> ii  samba-common-bin                 99:4.9.4-10  amd64 Glue package for
> sernet-samba-client.
> ii  sernet-samba                     99:4.9.4-10  amd64 SMB/CIFS file,
> print, and login server for Unix
> ii  sernet-samba-ad                  99:4.9.4-10  amd64 Samba Active
> Directory Domain Controller
> ii  sernet-samba-client              99:4.9.4-10  amd64        a
> LanManager-like simple client for Unix
> ii  sernet-samba-common              99:4.9.4-10  all Samba common files
> used by both the server and the client
> ii  sernet-samba-libs:amd64          99:4.9.4-10  amd64 Samba common
> library files used by both the server and the client
> ii  sernet-samba-libsmbclient0:amd64 99:4.9.4-10  amd64 Shared library
> that allows applications to talk to SMB servers
> ii  sernet-samba-winbind             99:4.9.4-10  amd64 Samba
> nameservice integration server
>
>
>
>

-- 
Denis Cardon
Tranquil IT Systems
Les Espaces Jules Verne, bâtiment A
12 avenue Jules Verne
44230 Saint Sébastien sur Loire
tel : +33 (0) 2.40.97.57.55
http://www.tranquil.it

Samba install wiki for Frenchies : https://dev.tranquil.it
WAPT, software deployment made easy : https://wapt.fr