CTDB Segfault in a container-based env - Looking for pointers

[Amitay Isaacs]

Hi Michael,

On Fri, Jul 16, 2021 at 5:47 PM Michael Adam <obnox at samba.org> wrote:
>
>
> Yes, as John and I have demonstrated in our sambaXP presentation,
>
> https://sambaxp.org/fileadmin/user_upload/sambaxp2021-slides/Mulligan_Adam_samba_operator.pdf
> https://www.youtube.com/watch?v=mG-Jxaf8_gw

The presentation provided me with the required background. :-)

> As Alexander explained in his mail, the “microservice” approach is a paradigm by which each container should be as simple as possible, ideally just encapsulating a single process, and containers interacting with each other over network and other interfaces, possible multiple containers bundled into a pod if needed and appropriate. One fundamental idea behind this is this: if the application / service is comprised of multiple components, and each component is isolated in a container image that is as minimal as possible, the whole management of the application like scaling of individual components etc can all be done by a generic container orchestration system, in our case kubernetes. This just works best if each container is minimal. Rescheduling a more complex, heavyweight container multiple server components in that same container is possibly much more disruptive and problematic.

This is the part I struggle with the most.  Even though you might
separate individual processes in containers (just because you can) and
group them as a pod, for me conceptually that pod (with multiple
containers) is providing the required service and like to think that
as a unit of abstraction.  You cannot deploy the service with only
partial pod (some of the containers), so every time you need to deploy
it's all the containers or nothing.  There might be an advantage to
separate the individual processes in containers, but I fail to see it.
If one container in a pod goes down, the whole pod is inactive because
it cannot provide the service.  In such a situation, just restarting
the failed container might not restart the service (it will depend on
the code that's running in the failed container).  From this point of
view, one needs to run a bunch of daemons as a single unit (whether
you call it a pod or I mis-refer to it as a container) to provide the
service.

>
> Since we are looking at managing samba as much as possible in a kubernetes/container-native way here (see the sambaXP preso), it is natural to aim at as much a micro-service approach as possible. We will certainly have to do some modifications to Samba / ctdb itself at some point to go the last mile, and I am convinced that this will be beneficial to the software as such, but of course the first approach is to see how far we can get without any modifications.

I would not like to call samba a microservice (may be in a scaling up
context with non-overlapping shares).  Clustered samba instance is
probably more like a microservice in the cloud native way of scaling
out (as you can spin up more instances to handle more load).  But
let's not get stuck in semantics. :-)

>
> But since you know what the real issue is, would you please enlighten us? :-)

The issue is that CTDB makes assumptions about the orphan processes.
On most unix systems, an orphan process gets re-parented to init which
traditionally has pid = 1.  This assumption is built into the code to
avoid runaway orhan processes in CTDB.

In the container world, what happens to orphan processes?

>
> Even if you don’t see a real benefit of this containerized layout just yet, it might still be beneficial for the code to consider some modifications to make ctdb more “container-ready”...

Provided it makes sense. ;-)

Amitay.