recent flapping tests

[Andrew Bartlett]

On Fri, 2019-06-07 at 22:42 +1200, Douglas Bagnall via samba-technical
wrote:
> Looking at the ones that flapped more than once, first on 14.04, then
> 18.04:
> 
> Now I'm switching to looking at all our 18.04 bugs since March 1,
> not just the ones that overlap with the final days of 14.04.
> 

Thank you so much for doing these.  It is really important to quantify
this problem space!

> 16 auth_log_winbind failures:
> 
>   10 UNEXPECTED(failure):
> samba.tests.auth_log_winbind.samba.tests.auth_log_winbind.AuthLogTest
> sWinbind.test_wbinfo(ad_member:local)
>    6 UNEXPECTED(failure):
> samba.tests.auth_log_winbind.samba.tests.auth_log_winbind.AuthLogTest
> sWinbind.test_wbinfo_ntlmv1(ad_member:local)

These should be fixed by Gary's patch
968434525480020999a30881ca20b9b13c99ad65 pushed by me on 6 June.


> The to 2019-02-22 numbers are looking at ~53 days or 212 test runs,
> while today's numbers look at ~99 days or 396 tests. Converting to
> percentage chance of failure per test:
> 
> wbinfo:             4.3
> notify:             4.0
> ctdb:               4.0
> auth_log_winbind:   4.0
> password_lockout:   2.3
> ldap.notification:  1.3
> audit:              1.3
> rpc:                1.3
> others:             1.5
> in total:           24
> 
> That is, our flakey tests have been failing 24% of the time. In the
> previous time period, they failed 17.5% of the time, so it is
> unfortunately true that we made things worse.

Hopefully with the above and the CTDB fixes from Martin we are now down
to 16% again, but help on the notify tests would be a good way to knock
another 4% off that.  I think a goal of only failing autobuild 1/10
runs is achievable. 

Andrew Bartlett

-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba