Need for cleanup of idle winbindd child processes...

[Ravindra Channabasappa]

Volker,
  
    In my setup there are only two winbindd processes. One parent winbindd and the other one for the configured domain.
I have not enabled trusted domain configuration.  Do you still think mmap'ing tdb files has issues?  But I observe that for every run of 5000 users login, RSS size increases by ~500 KB (ps -eo output) and never comes down.

I tried checking through 'smbcontrol --pool-usage' command. Output of the same is attached for before the load test and after the test.
I could not figure much difference between the outputs.  About valgrind output- having problem in getting valgrind run under jail environment where winbindd is running.

Also, in another thread - "RE: [PATCH] Unix datagram socket messaging"- I see that patch mentions about cleanup of winbindd processes as well.
If so, IMO based on internal testing 15 minutes to cleanup child processes looks long. Is it fine to reduce the time to 5 minutes or make this configurable through smb.conf ?
I hope we can apply this patch for Samba-4.1.3...

If the above said patch does not apply for cleanup of winbindd then based on earlier discussion would like propose the attached patch for winbindd cleanup.

Thanks,
--ravindra






-----Original Message-----
From: Volker Lendecke [mailto:Volker.Lendecke at SerNet.DE] 
Sent: Monday, March 31, 2014 2:29 PM
To: Jeremy Allison
Cc: samba-technical at lists.samba.org; Ravindra Channabasappa
Subject: Re: Need for cleanup of idle winbindd child processes...

On Thu, Mar 27, 2014 at 11:28:19AM -0700, Jeremy Allison wrote:
> On Thu, Mar 27, 2014 at 03:56:32PM +0100, David Disseldorp wrote:
> > Hi Ravindra,
> > 
> > On Thu, 27 Mar 2014 14:25:01 +0000, Ravindra Channabasappa wrote:
> > 
> > > If we enable trusted domain flag and there are 10 trusted domains, the number of processes increase (10x10) under load. Once started these processes do not exit. Also, it is observed in our setup that the memory consumed by these child processes keeps on increasing (for every load test run) without any sign of reducing. This has performance implications as we successively run load test on the winbindd daemon.
> > 
> > Do you have any "smbcontrol <pid> pool-usage" dumps, or valgrind 
> > logs showing the memory leak?
> 
> Yep. Fixing the memory leak would be preferable :-).
> Finding it comes first..

Indeed.

One thing that might contribute to the memory shown in "top"
or equivalent tools is the tdb files winbind uses, most notably the winbindd_cache.tdb and the gencache files. Those are mmapp'ed by all winbind processes, thus shared. In my experience it is very hard to correctly interpret the RSS output in top in the light of large shared memory mappings.

Just for testing purposes, you might want to set

use mmap = no

which avoids memory mapping of tdbs at a significant performance penaltly. But it will tell you whether you are seeing a genuine winbind memory leak or whether the RSS increase is due to large tdb files.

With best regards,

Volker Lendecke

--
SerNet GmbH, Bahnhofsallee 1b, 37081 Göttingen
phone: +49-551-370000-0, fax: +49-551-370000-9 AG Göttingen, HRB 2816, GF: Dr. Johannes Loxen http://www.sernet.de, mailto:kontakt at sernet.de


-------------- next part --------------
A non-text attachment was scrubbed...
Name: exit_child_3.patch
Type: application/octet-stream
Size: 1925 bytes
Desc: exit_child_3.patch
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20140421/1bb0d636/attachment.obj>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: smb-control-after.txt
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20140421/1bb0d636/attachment.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: smb-control-before.txt
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20140421/1bb0d636/attachment-0001.txt>