smbd notifyd daemon memory leaking

[Youzhong Yang]

Hello folks,

We have Samba 4.9.4 running on a bunch of servers, occasionally on some servers, the notifyd daemon process started to use lots of memory. Yesterday on one of our servers, it used 270G of memory.

I ran dtrace on the smbd/notifyd process, as you can see, the memory allocation calls are far more than memory free calls.

Does this ring a bell to you? Is this a known issue?

Thanks for your attention,

-yyang

# dtrace -n 'pid$target::umem_alloc:entry,pid$target::umem_free:entry {@[ustack()] = count();} tick-60s {exit(0);}' -p 6369
dtrace: description 'pid$target::umem_alloc:entry,pid$target::umem_free:entry ' matched 3 probes
CPU     ID                    FUNCTION:NAME
12  94612                        :tick-60s


              libumem.so.1`umem_free
              libumem.so.1`process_free+0x145
              libumem.so.1`umem_malloc_free+0x1d
              libtalloc.so.2.1.14`_tc_free_internal+0x3ea
              libtalloc.so.2.1.14`_talloc_free_internal+0x9d
              libtalloc.so.2.1.14`_talloc_free+0x105
              libsmbconf.so.0`messaging_post_main_event_context+0x1c7
              libtevent.so.0.9.37`tevent_common_invoke_immediate_handler+0x184
              libtevent.so.0.9.37`tevent_common_loop_immediate+0x37
              libtevent.so.0.9.37`epoll_event_loop_once+0x8f
              libtevent.so.0.9.37`std_event_loop_once+0x5c
              libtevent.so.0.9.37`_tevent_loop_once+0x102
              libtevent.so.0.9.37`tevent_req_poll+0x25
              smbd`smbd_notifyd_init+0x28c
              smbd`main+0xf67
              smbd`_start_crt+0x83
              smbd`_start+0x18
                3

              libumem.so.1`umem_alloc
              libumem.so.1`umem_malloc+0x3f
              libtalloc.so.2.1.14`__talloc_with_prefix+0x138
              libtalloc.so.2.1.14`_talloc_pool+0x2c
              libtalloc.so.2.1.14`_talloc_pooled_object+0xc1
              libsmbconf.so.0`messaging_rec_dup+0x72
              libsmbconf.so.0`messaging_rec_create+0x228
              libsmbconf.so.0`messaging_post_self+0x54
              libsmbconf.so.0`messaging_send_iov_from+0x11f
              libsmbd-base-samba4.so`notifyd_send_delete+0xed
              libsmbd-base-samba4.so`notifyd_trigger_parser+0x3b9
              libdbwrap-samba4.so`db_rbt_parse_record+0x76
              libdbwrap-samba4.so`dbwrap_parse_record+0x55
              libsmbd-base-samba4.so`notifyd_trigger+0x3c2
              libsmbconf.so.0`messaging_dispatch_classic+0xda
              libsmbconf.so.0`messaging_post_main_event_context+0x13c
              libtevent.so.0.9.37`tevent_common_invoke_immediate_handler+0x184
              libtevent.so.0.9.37`tevent_common_loop_immediate+0x37
              libtevent.so.0.9.37`epoll_event_loop_once+0x8f
              libtevent.so.0.9.37`std_event_loop_once+0x5c
             9163