[Samba] How to profile a samba function

[Andrew Bartlett]

On Thu, 2013-10-17 at 15:18 +0100, Alex Matthews wrote:
> Hi all,
> 
> I am trying to track down what is causing the slow running of 
> sysvolreset on my system.
> I am aware that I have a great deal of files in my 'netlogon' directory 
> which is why the problem is so apparent for me.
> 
> sysvolreset takes approx 5+ minutes to complete.
> 
> Tracing out samba-tool shows that the slowdown is caused by the samba 
> function 'set_nt_acl'.
> This function takes approx 120ms to run, which when multiplied up by a 
> few thousand files (2,000 in my case) you very quickly end up in minutes.
> 
> running samba-tool using cProfile ('python2.7 -m cProfile 
> /usr/bin/samba-tool ntacl sysvolreset') gives the following results: 
> http://pastebin.com/98rYXRxe
> 
> As you can see the only call that takes any 'significant' time is 
> "samba.samba3.smbd.set_nt_acl".
> Because this function is part of the samba source rather than part of a 
> python script cProfile cannot profile any further.
> How can I profile this function in smbd (without having to recompile 
> samba would be ideal).
> I've seen 'perf' mentioned in other threads, is there any way this tool 
> can be used?
> If I have to recompile and add my own timing code I will, but only 
> really as a last resort.
> 
> This problem can't be isolated to just sysvolreset as the 'set_nt_acl' 
> function must be called quite frequently elsewhere.
> I think 120ms for a call to return is quite a long time. I would expect 
> to see an overall performance gain if it can be optimised a bit!

Perf is ideal for this, and has quickly become my favourite profiling
tool, because it's the first one I've found that I can just pick up and
use!.  'perf record -g /usr/bin/samba-tool ntacl sysvolreset' and then
'perf report -g' will give a lot of really useful output, in particular
call graphs to isolate not only where the time is spent, but who is
requesting the time be spent. 

I've CC'ed Nikos, who needs help profiling samba-tool dbcheck.  I wonder
if you could help each other out with the profiling? 

Thanks,

Andrew Bartlett

-- 
Andrew Bartlett
http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org
Samba Developer, Catalyst IT                   http://catalyst.net.nz