samba_dnsupdate timeouts (was Re: [PATCH] python indent bugfix in dns_hub.py)
metze at samba.org
Tue Feb 5 08:13:13 UTC 2019
Am 05.02.19 um 05:27 schrieb Tim Beale:
> Yeah, it looks like changing the process model was enough to push the CI
> runners over the edge fairly reliably.
> I tried out a couple of ideas: reducing the prefork workers, reordering
> the tests, tweaking the timeouts, etc. But it seems like the swap
> problem just manifests itself elsewhere.
> Attached is a patch that splits up the autobuild job. I wasn't sure if
> we needed a py2 target as well, or whether we were dropping py2 support
> from 4.11.
I think we should at it for now and then backport it to 4.10.
> CI links:
> - https://gitlab.com/samba-team/devel/samba/pipelines/46145771 (result
> looks OK, but I had forgotten to update builddirs in the patch)
> - https://gitlab.com/samba-team/devel/samba/pipelines/46152211 (just
> kicked it off again now)
It's still running, but as it seems to be almost imposible to get a
pipeline through without it, I have pushed it to autobuild already.
> On 2/02/19 7:42 AM, Andrew Bartlett wrote:
>> On Fri, 2019-02-01 at 16:20 +0100, Stefan Metzmacher wrote:
>>> Can this be a really busy gitlab shared runner that is swapping?
>> I was thinking something like that.
>> Some of the shared runners are quite resource constrained, and any
>> python script that imports Samba's 'samba' python package will load
>> quite a bit of code due to the size of our bindings.
>> Looking at the system-info.txt:
>> ### free
>> total used free shared buffers cach
>> Mem: 3784700 685648 3099052 169564 8636 3079
>> -/+ buffers/cache: 369072 3415628
>> Swap: 8388600 166420 8222180
>> This is after it finishes, but I would suggest that having pushed into
>> swap at all it is likely that the < 4GB of ram has already been
>>> Does anyone has any idea?
>> I asked Joe to see if the imports could be made faster.
>>> Maybe giving samba_dnsupdate a 60 second timeout would be a workaround?
>> That could work, but I think the swap storm is likely to just hit
>> something else.
>> We can also split up samba-ad-dc-2 into samba-ad-dc-3 by putting the
>> backup and restore DCs tests in their own autobuild jobs. This would
>> make sn-devel more busy (more complies) but be cheap on the shared
>> runners (free CI).
>> Or we could rework selftest.pl to sort jobs by environment and shut
>> down the environment after the last job that needs it (as determined by
>> the dep tree we now have).
>> How does that sound for medium and long-term plans?
>> Finally, I did merge Tim's patch to change the default process model,
>> so that will have changed the memory behaviour a lot.
>> Andrew Bartlett
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 833 bytes
Desc: OpenPGP digital signature
More information about the samba-technical