ctdb: Adding memory pool for queue callback

[Martin Schwenke]

Hi Swen,

On Mon, 20 Aug 2018 09:56:02 +0200, Swen Schillig <swen at vnet.ibm.com>
wrote:

> This is an updated version of the patch I sent out a few weeks ago.
> 
> The reason for having an update was that further tests showed,
> that it was not beneficial to have the queue_send routine being a
> consumer of the pool as well.
> In our tests we saw that it is not likely that additional memory needs
> to be allocated, but if it is, the size is so big that all of the pool
> memory is used and kept for a considerably long time.
> This has the negative side effect on the other consumer
> (receiver / queue_read) which can not benefit from the pool anymore
> until this memory chunk is free'd again.
> 
> Please review and push if happy.

This is generally a good improvement.

I think the following part of the commit message probably needs to be
updated:

  All received messages are processed synchronously per queue,
  therefore, a larger memory pool is not requried.

There are definitely packets that are processed asynchronously.  Any of
the control handling functions called by ctdb_control_dispatch() that
have async_reply passed to them can talloc_steal() the control structure
(i.e. the packet, because structs on the wire!).  Similarly for things
like ctdb_reply_call().

However, even with some packets processed asynchronously I think that
this should provide a useful reduction in malloc()s.

Can you please take a look at these code paths and, if convinced,
update the commit message?

Thanks...

peace & happiness,
martin