[Samba] Samba 4.9.4 - high RAM usage - OOM killer

[Laurent CARON]

Hi,


Swappiness is indeed set to 60 (default).


The DCs don't have any swap.


Do you think I really need 16GB RAM for such a setup ?


Thanks

On 28/01/2019 12:08, L.P.H. van Belle via samba wrote:
> Hai,
>
> How is the swappiness set?
>
> cat /proc/sys/vm/swappiness
> Its probely set to 60, i suggest, lower it to 10 or 20.
>
> That wil help keeping the samba processes out of swap.
> I've see more programs gettting OOM Killed due too out of swap.
>
> echo 10 > /proc/sys/vm/swappiness
> Stop and start samba
>
>
> Greetz,
>
> Louis
>
>
>
>
>> -----Oorspronkelijk bericht-----
>> Van: samba [mailto:samba-bounces at lists.samba.org] Namens
>> Laurent CARON via samba
>> Verzonden: maandag 28 januari 2019 11:46
>> Aan: samba at lists.samba.org
>> Onderwerp: [Samba] Samba 4.9.4 - high RAM usage - OOM killer
>>
>> Hi,
>>
>>
>> We upgraded a legacy (NT4) domain from 3.6 series to 4.8 and
>> then 4.9.4
>> samba version (using sernet subscription packages / debian stable)
>>
>> The setup is composed of 4 DCs with each 2 CPU/16GB RAM.
>>
>> We currently have ~700 user accounts / ~600 computers / ~150 groups
>>
>> Our mail setup, SSO, ... query the 4 DCs constantly.
>>
>>
>> Every 5 to 10 days the RAM consumption and CPU usage (due to
>> kswapd) are
>> peaking.
>>
>> This leads to OOM killer killing samba processes
>>
>> kernel: [765104.826327] samba invoked oom-killer:
>> gfp_mask=0x24201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0,
>> order=0, oom_score_adj=0
>> kernel: [765104.826355]  [<ffffffff8c3871ba>] ?
>> oom_kill_process+0x21a/0x3e0
>> kernel: [765104.826357]  [<ffffffff8c386e3d>] ? oom_badness+0xed/0x170
>> kernel: [765104.826455] [ pid ]   uid  tgid total_vm      rss nr_ptes
>> nr_pmds swapents oom_score_adj name
>>
>> ...
>>
>> kernel: [861216.518771] Out of memory: Kill process 603
>> (samba) score 3
>> or sacrifice child
>> kernel: [861356.048484]  [<ffffffff8c387651>] ?
>> out_of_memory+0x111/0x470
>>
>> samba[614]:   ../source4/dsdb/kcc/kcc_periodic.c:768: Failed
>> samba_kcc -
>> NT_STATUS_NO_MEMORY
>>
>> Once this happens, the affected DC is unresponsive for all samba
>> authentication processes (including LDAP).
>>
>> A reboot of the affected VM 'cures' the issue, but only for a short
>> amount of time (5 to 10 days).
>>
>> Apart from either restarting samba processes on a daily basis, or
>> rebooting the DCs, is there a way to:
>>
>> - pinpoint the root cause of the memory consumption (leak,
>> corrupted DB,
>> ...)
>>
>> - have the DCs use a more 'normal' amount of RAM ?
>>
>>
>> Thanks
>>
>>
>>
>> Please note:
>>
>> # samba-tool drs kcc
>>
>> # samba-tool dbcheck --cross-ncs
>>
>> are not showing any errors
>>
>>
>> :/etc/nsswitch.conf:
>>
>> passwd:         compat
>> group:          compat
>> shadow:         compat
>> gshadow:        files
>>
>> hosts:          files dns
>> networks:       files
>>
>> protocols:      db files
>> services:       db files
>> ethers:         db files
>> rpc:            db files
>>
>>
>> :/etc/samba/smb.conf:
>>
>> [global]
>>           netbios name = VS-DC-001
>>           realm = CORP.MYDOMAIN
>>           workgroup = SAMBA
>>
>>           log file = /var/log/samba/samba.log.%m
>>           log level = 1 auth_audit:3 auth_json_audit:3
>>           max log size = 50000
>>           debug timestamp = yes
>>           dns forwarder = 192.168.0.20 192.168.100.20 192.168.0.21
>> 192.168.100.21
>>           server role check:inhibit=yes
>>           ldap server require strong auth = no
>>           wins support = yes
>>           server role = active directory domain controller
>>           check password script = /usr/local/bin/crackcheck -c -d
>> /var/cache/cracklib/cracklib_dict
>>           idmap_ldb:use rfc2307 = yes
>>           server schannel = auto
>>
>> [netlogon]
>>           path = /var/lib/samba/sysvol/corp.lncsa.com/scripts
>>           read only = No
>>
>> [sysvol]
>>           path = /var/lib/samba/sysvol
>>           read only = No
>>
>> # du -shxc sam.ldb*
>> 4.1M    sam.ldb
>> 132M    sam.ldb.d
>> 136M    total
>>
>> Samba packages:
>>
>> ii  samba                            99:4.9.4-10  amd64 Glue
>> package for
>> sernet-samba.
>> ii  samba-common                     99:4.9.4-10  all Glue
>> package for
>> sernet-samba-common.
>> ii  samba-common-bin                 99:4.9.4-10  amd64 Glue
>> package for
>> sernet-samba-client.
>> ii  sernet-samba                     99:4.9.4-10  amd64
>> SMB/CIFS file,
>> print, and login server for Unix
>> ii  sernet-samba-ad                  99:4.9.4-10  amd64 Samba Active
>> Directory Domain Controller
>> ii  sernet-samba-client              99:4.9.4-10  amd64        a
>> LanManager-like simple client for Unix
>> ii  sernet-samba-common              99:4.9.4-10  all Samba
>> common files
>> used by both the server and the client
>> ii  sernet-samba-libs:amd64          99:4.9.4-10  amd64 Samba common
>> library files used by both the server and the client
>> ii  sernet-samba-libsmbclient0:amd64 99:4.9.4-10  amd64
>> Shared library
>> that allows applications to talk to SMB servers
>> ii  sernet-samba-winbind             99:4.9.4-10  amd64 Samba
>> nameservice integration server
>>
>>
>>
>>
>> -- 
>> To unsubscribe from this list go to the following URL and read the
>> instructions:  https://lists.samba.org/mailman/options/samba
>>
>