[RFC] [CTDB] Optimized memory handling.

[Swen Schillig]

On Thu, 2017-12-14 at 10:42 +1100, Amitay Isaacs via samba-technical
wrote:
> On Wed, Dec 13, 2017 at 10:52 AM, Jeremy Allison via samba-technical
> <samba-technical at lists.samba.org> wrote:
> > On Tue, Dec 12, 2017 at 12:49:08PM +0100, Swen Schillig via samba-
> > technical wrote:
> > > ...
> > > The currently used magic numbers for the pool sizes are still
> > > under
> > > investigation...hints and recommendations are welcome.
> > > 
> > > However, initial tests showed big improvements.
> > 
> > Can you give more details on what you mean by "big improvements" ?
> > Some numbers would be good :-).
> > 
> > Thanks,
> > 
> > Jeremy.
> > 
> 
> Even without seeing the proof I can imagine the improvements in
> packet
> handling using the talloc pool.  That code is in the hot path and
> avoiding extra malloc() calls will definitely result in reducing the
> load on ctdb daemon.  However, as Jeremy mentioned, it would be good
> to verify with real numbers.
> ...
> 
> Amitay.
> 
Thank you guys for having a look, it's really appreciated.

As for the numbers, I do not have any "real" numbers yet.
But an isolated test of the code in question did show improvements of
approx. 40% of pool-alloc-code vs. non-pool-alloc-code.
This means just for the areas in question and not running a 
real-life load.
Anyhow, as Amitay stated, this is the hot path and if we're "only"
getting 5% in the end that would be "a big improvement" :-)

I'm in the process of getting a cluster ready to perform 
real-life-tests to get an idea of some good initial pool-size values,
(especially for the data pool).

@Amitay: I will try and transfer the changes to sock_io today so you
guys can have a look again.... making sure the direction is right.

Regarding the "static" data pool. 
I'm not a fan of those either, but I couldn't find a decent context to
attach the pool to. Therefore, I decided to setup "one" pool once
for all data-buffer requirements instead of being dependent on some
upper structure which is not related in any way to the data mem.

Cheers Swen