samba_dnsupdate timeouts (was Re: [PATCH] python indent bugfix in dns_hub.py)

[Stefan Metzmacher]

Hi Tim,

Am 05.02.19 um 05:27 schrieb Tim Beale:
> Yeah, it looks like changing the process model was enough to push the CI
> runners over the edge fairly reliably.
> 
> I tried out a couple of ideas: reducing the prefork workers, reordering
> the tests, tweaking the timeouts, etc. But it seems like the swap
> problem just manifests itself elsewhere.
> 
> Attached is a patch that splits up the autobuild job. I wasn't sure if
> we needed a py2 target as well, or whether we were dropping py2 support
> from 4.11.

I think we should at it for now and then backport it to 4.10.

> CI links:
> - https://gitlab.com/samba-team/devel/samba/pipelines/46145771  (result
> looks OK, but I had forgotten to update builddirs in the patch)
> - https://gitlab.com/samba-team/devel/samba/pipelines/46152211  (just
> kicked it off again now)

It's still running, but as it seems to be almost imposible to get a
pipeline through without it, I have pushed it to autobuild already.

Thanks!
metze

> On 2/02/19 7:42 AM, Andrew Bartlett wrote:
>> On Fri, 2019-02-01 at 16:20 +0100, Stefan Metzmacher wrote:
>>> Hi,
>>> Can this be a really busy gitlab shared runner that is swapping?
>> I was thinking something like that.  
>>
>> Some of the shared runners are quite resource constrained, and any
>> python script that imports Samba's 'samba' python package will load
>> quite a bit of code due to the size of our bindings. 
>>
>> Looking at the system-info.txt:
>>
>> ### free
>>              total       used       free     shared    buffers     cach
>> ed
>> Mem:       3784700     685648    3099052     169564       8636     3079
>> 40
>> -/+ buffers/cache:     369072    3415628
>> Swap:      8388600     166420    8222180
>>
>>
>> This is after it finishes, but I would suggest that having pushed into
>> swap at all it is likely that the < 4GB of ram has already been
>> consumed.
>>
>>> Does anyone has any idea?
>> I asked Joe to see if the imports could be made faster.
>>
>>> Maybe giving samba_dnsupdate a 60 second timeout would be a workaround?
>> That could work, but I think the swap storm is likely to just hit
>> something else.
>>
>> We can also split up samba-ad-dc-2 into samba-ad-dc-3 by putting the
>> backup and restore DCs tests in their own autobuild jobs.  This would
>> make sn-devel more busy (more complies) but be cheap on the shared
>> runners (free CI). 
>>
>> Or we could rework selftest.pl to sort jobs by environment and shut
>> down the environment after the last job that needs it (as determined by
>> the dep tree we now have).  
>>
>> How does that sound for medium and long-term plans?
>>
>> Finally, I did merge Tim's patch to change the default process model,
>> so that will have changed the memory behaviour a lot. 
>>
>> Andrew Bartlett


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20190205/f169e958/signature.sig>