Samba4 and memory consumption - needing frequent kills

[Ricky Nance]

Kev, I think Andrew has patched it to where the leaks are either non
existant, or just very very slow. Currently using ps_mem.py I have this:

 51.1 MiB +   2.4 MiB =  53.5 MiB       named
100.9 MiB +  50.0 MiB = 151.0 MiB       samba (13)
233.7 MiB +  43.0 MiB = 276.8 MiB       smbd (36)

It should be noted that I am using s3fs and bind9 dlz in my setup and have
around 350 users and about 250 computers. I will also say that the named
process hasn't gained in size like it was. This setup has been running for
about 3 days now, and memory usage seems fine. I am running Version
4.0.0beta7-GIT-4f4bb1f, as of the writing of this email its 7:20 on a the
8/21 and this was built around 21:00 on 8/17 if that helps give you an
estimate of time. How exactly are you checking memory consumption on your
setup?

Ricky

On Tue, Aug 21, 2012 at 3:39 AM, Kev Latimer <klatimer at tolent.co.uk> wrote:

> On 15/08/2012 03:30, Ricky Nance wrote:
>
>> Glad I could help.
>> Thank you guys for all the hard work you have done on this!
>>
>> Ricky
>>
>> On Tue, Aug 14, 2012 at 9:22 PM, Andrew Bartlett <abartlet at samba.org>
>> wrote:
>>
>>  On Sat, 2012-08-11 at 17:00 +1000, Andrew Bartlett wrote:
>>>
>>>> On Thu, 2012-08-09 at 17:08 +1000, Andrew Bartlett wrote:
>>>>
>>>>> On Wed, 2012-08-08 at 21:04 +0200, Arvid Requate wrote:
>>>>>
>>>>>> Hello Kev,
>>>>>>
>>>>>> maybe you are facing the same issue here as reported in
>>>>>> https://bugzilla.samba.org/**show_bug.cgi?id=8827<https://bugzilla.samba.org/show_bug.cgi?id=8827>?
>>>>>>
>>>>> As I've just commented on the bug, I need a --leak-report-full rather
>>>>> than just --leak-report.
>>>>>
>>>>> Also, if that does not show the cause, perhaps instead of shutting down
>>>>> the server run with --leak-report-full, then attach to the large
>>>>>
>>>> process
>>>
>>>> with:
>>>>>
>>>> Given the issues with the memory dumps being repeated in overwealming
>>>> detail, here is a new instruction:
>>>>
>>>> Run samba as:
>>>>
>>>> gdb --args samba -M single --leak-report
>>>>
>>>> Then:
>>>>
>>>> run
>>>>
>>>> When it becomes large, run
>>>>
>>>> p talloc_report_full(0, stderr)
>>>>
>>>> and mail me *only* the output of that command, not the earlier or later
>>>>
>>> output.
>>>
>>>> The issue with the previous logs is that the same allocation (present in
>>>> multiple forked children) was being logged multiple times, as each child
>>>> exited.  This caused the massive log buildup, without giving me much
>>>> more information.
>>>>
>>>> You may replace stderr with a file on disk with something like:
>>>>
>>>> p talloc_report_full(0, fopen("/tmp/leak.txt", "w")
>>>>
>>>> Hopefully this will give you and me the information we need.
>>>>
>>> Thanks to Ricky Nace who was able to provide me with exactly this (on a
>>> standard process modal run it turns out), I've been able to fix this.
>>>
>>> For sites running older Samba betas, this patch series might be helpful.
>>> Specifically, it should be tested and worked into things like the debian
>>> packages, as they are stuck at beta2.
>>>
>>> Andrew Bartlett
>>>
>>> --
>>> Andrew Bartlett
>>> http://samba.org/~abartlet/
>>> Authentication Developer, Samba Team           http://samba.org
>>>
>>>
>>>
>> --
>>
> Hi Andrew,
>
> Just wanted to quickly apologise for not getting all the info you needed
> as quickly as either of us would have liked.  My availability was zero last
> week because of an internal project and I simply wasn't able to run my
> samba install in the way you needed as I wasn't there to keep any kind of
> eye on it.  I did find a few hours on Thursday night, after I'd seen this
> mail, to grab the latest master and I can confirm the issue has certainly
> been mitigated although I'm not sure it's gone completely.  I've been
> running this build since the 16th, so the fact it hasn't devoured all my
> swap yet is certainly progress, but I do have a few processes larger than
> the others still, just not massively larger: two at around 750MB, one
> tipping the scales at the wrong side of 1GB.  I'm running
> 4.0.0beta7-GIT-56fc7bc.
>
> Would you be able to confirm if your fixes are in the git I'm running and
> if that consumption is to be expected?  I'm more than happy to produce the
> reports you were after if they're still of any use if you think there's
> anything else worth looking at.
>
> Ricky - are you able to report back on your results?
>
> Thanks again for all your efforts.
>
> Cheers,
>
> Kev
>
> --
> Kev
>



--