[PATCH] Avoid using CTDB_BROADCAST_ALL (bug 13056)

[Amitay Isaacs]

On Fri, Sep 29, 2017 at 12:45 PM, Amitay Isaacs <amitay at gmail.com> wrote:

> On Thu, Sep 28, 2017 at 9:17 PM, Volker Lendecke <
> Volker.Lendecke at sernet.de> wrote:
>
>> On Thu, Sep 28, 2017 at 03:19:28PM +1000, Amitay Isaacs via
>> samba-technical wrote:
>> > Hi,
>> >
>> > CTDB in some places uses CTDB_BROADCAST_ALL to
>> > distribute packets to all the configured nodes.  If there is a
>> > dead node (or a node that is not running) in the cluster,
>> > then ctdb daemons will queue up packets for such nodes.
>> > Over long periods of time, ctdb daemon can consume lots
>> > of memory which will not be freed till the dead node is
>> > started or ctdb daemon is restarted.
>> >
>> > Use CTDB_BROADCAST_CONNECTED instead to send
>> > packets to the nodes that are running.
>>
>> Pushed, thanks!uu
>>
>> Volker
>>
>
> Turns out that the fix is not as trivial as I thought earlier.
> Sorry for the trouble.
>
> Just replacing all BROADCAST_ALL with BROADCAST_CONNECTED
> introduces all sorts of race conditions during ctdb startup.  Some of the
> messages (e.g. election) have to be sent using BROADCAST_ALL.
> I need to fix the logic for handling BROADCAST_ALL to not queue the
> packets for dead nodes.
>
>
Here are the updated patches.

- Replace BROADCAST_ALL with BROADCAST_CONNECTED for
  database controls
- Do not queue packet if the queue fd is invalid

BROADCAST_ALL is still used in following two cases to avoid
the race conditions during ctdb startup.
- Election
- Distributing tickles

Please review and push.

Amitay.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ctdb.patch
Type: text/x-patch
Size: 3820 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20171003/2c63a28d/ctdb.bin>