samba profiling and pcp pmda

[James Peach]

On 11/04/2007, at 6:34 PM, Alison Winters wrote:

> CC:ed into samba-technical from personal email, this is a discussion
> about the steps we could to take to enhance the existing profiling  
> code
> in Samba.
>
> James Peach wrote:
>> The metrics export mechanism from jpeach-xmetrics belongs in the main
>> samba tree.
>>
> Agreed.
>
>> The first thing you need to do to get this in is to split it into
>> multiple mmap'ed files. That is, each different metrics structure  
>> should
>> have its own mmap and backing store. Once you do this, the allocator
>> becomes a simple slot allocator and most of the really ugly code  
>> goes away.
>>
> Do you mean having one header file that everything mmaps and a  
> bunch of
> files for each connected client and configured share that get mmapped
> separately?

I mean to use one mmap per record type. The current scheme is ugly  
because it really doesn't want to be a proper allocator but it has to  
deal with 2 different sizes of records in the mmap. If you have a  
separate mmap for each record type, the allocator goes back to being  
nice and simple.

> I'm assuming the ugly code you're referring to is the bit
> where it has to keep growing the file whenever new connections join  
> etc.

No, I'm referring to the allocation of metric records inside the mmap.

>
>> The second thing that you need os some way of gathering the stats  
>> that
>> is not that horrible VFS module :)
>>
> That's in the pipeline :)  What i'm planning is to just add a bit of
> extra magic to the existing Samba profiling macros that allow us to
> split it out per-client and per-share along with the existing  
> recording
> of global counts and times.  I'm pretty sure this is going to be  
> fairly
> elegant and small if i get it right.

That sounds neat. It would be great if you had some code to reconcile  
the existing profiling stats with these metrics.

> I have a separate patch which i can't submit yet (it's copyright SGI)
> that splits out all of the current profiling statistics into smaller
> groups that we can turn on and off independently.

Do you mean the stats that currently live in the sysv shared memory  
segment?

> This will allow us
> more granular control over what to profile, and will hopefully avoid
> nasty performance hits when we're only interested in isolated areas of
> the code.

IIRC you only get a big performance hit when you are timing  
operations. You can probably avoid this (to some degree) by using  
platform-specific timers.

>> Once you have a stable, versioned export mechanism, then the PMDA can
>> simply be one of the consumers of the exported metrics. There's  
>> not real
>> need for it to be in the Samba tree. I'd be OK with making the client
>> interfaces (header file and shared library) LGPL.
>>
> I guess my real problem with this is that there's already a PMDA in  
> the
> pcp/ subdirectory in Samba.  This PMDA is broken right now - it won't
> compile because some definitions have been moved around.  If it's not
> going to be maintained then it should be removed from the tree - i can
> just have a closed-source PMDA that builds independently (assuming  
> LGPL
> headers).  On the other hand, since it's in the tree anyway it  
> could be
> maintained as part of Samba - in which case i think applying the
> attached two patches (move, then add) would be the right thing to do.
> These patches move the PMDA into the main build - they're just a  
> subset
> of your xmetrics branch.

IMHO, the Samba PMDA belongs in the PCP tree, not in the Samba tree.  
Since you are going to be the one maintaining it, it's easier all  
round if it lives there. We can simply remove the existing PMDA from  
the Samba tree.

--
James Peach | jpeach at samba.org