samba_dnsupdate timeouts (was Re: [PATCH] python indent bugfix in dns_hub.py)

[Tim Beale]

I just wanted to say I think dns_hub has been a good addition to the
selftest framework. Anything that allows us to test DNS more
realistically is a good thing, and is worth a few teething problems.

I'm the one to blame here for breaking CI with my change to use the
prefork process model. I'm sorry for the hassle this caused everyone.

It at least highlighted the real problem, which was that we were
starting to hit the CI runner limits, and that could've dragged on for
months with CI failing intermittently for no obvious reason. For the
record, the CI limit seems to be around 8 DCs, although obviously this
varies somewhat depending on the process model overhead. Tearing down
testenvs once we're done with them seems like a good idea in the long run.

I'll raise a bug and backport the autobuild change to 4.10.

On 5/02/19 8:54 PM, Jeremy Allison wrote:
> On Tue, Feb 05, 2019 at 08:44:08AM +0100, Volker Lendecke wrote:
>> On Tue, Feb 05, 2019 at 09:21:58AM +0200, Isaac Boukris via samba-technical wrote:
>>> Hi,
>>>
>>> On Tue, Feb 5, 2019 at 6:28 AM Tim Beale <timbeale at catalyst.net.nz> wrote:
>>>> Yeah, it looks like changing the process model was enough to push the CI
>>>> runners over the edge fairly reliably.
>>> FYI, I just tried to rebase merge request !200 on master and
>>> build_samba_ad_dc_2 failed on a somewhat different dns error:
>>> https://gitlab.com/samba-team/devel/samba/-/jobs/156308654
>> I'll work on a patchset to restore the pre-async-dns/dns_hub behaviour
>> and remove the dns_hub again. We need to find a different way to
>> approach async DNS and properly test it. The current approach has
>> proven to be the wrong way.
> Please don't do that. It appears to be a python / resource limit
> issue on the gitlab-CI runners. Removing dns_hub isn't going
> to make *any* difference here, we're going to run into this
> again and again until we find a way to break up the tests or
> get more resources on gitlab.
>
> Revert frenzy is never a good move. Don't do it please, it's
> a waste of time.