recent flapping tests

Fri Jun 7 10:42:49 UTC 2019

For a while this year we ran tests on both Ubuntu 14.04 and 18.04.
Comparing the two might reveal tests whose flappingness is sensitive
to the underlying environment.

Looking at the ones that flapped more than once, first on 14.04, then
18.04:

$ ./parse-email/parse-autobuild-email --file-regex sn-devel-144 --since
2019-03-01 --until 2019-04-26
found 44 lines matching '(?:^(?:UNEXPECTED|\\*FAILED\\*))|(?:No space left
on device)' in 52 files matching 'sn-devel-144'
   6 UNEXPECTED(failure):
samba.wbinfo_simple.check-secret.domain=SAMBA-TEST.wbinfo(nt4_member:local)
   5 UNEXPECTED(failure): samba3.smb2.notify.valid-req(nt4_dc)
   5 UNEXPECTED(failure):
samba4.ldap.notification.python(ad_dc_ntvfs).__main__.LDAPNotificationTest.test_simple_search(ad_dc_ntvfs)
   4 *FAILED* tests/simple/18_ctdb_reloadips.sh
   3 UNEXPECTED(failure): samba3.smb2.notify.mask(nt4_dc)
   2 UNEXPECTED(failure): samba3.raw.notify.mask(nt4_dc)
   2 *FAILED* tests/simple/60_recoverd_missing_ip.sh
   2 UNEXPECTED(failure):
samba3.smb2.notify-inotify.inotify-rename(fileserver)
   2 UNEXPECTED(failure):
samba4.winbind.struct.lookup_name_sid(nt4_member:local)
   2 UNEXPECTED(failure):
samba.tests.samba_tool.user_wdigest.samba.tests.samba_tool.user_wdigest.UserCmdWdigestTestCase.test_Wdigest01(ad_dc_default:local)

$ ./parse-email/parse-autobuild-email --file-regex sn-devel-184 --since
2019-03-01 --until 2019-04-26
found 47 lines matching '(?:^(?:UNEXPECTED|\\*FAILED\\*))|(?:No space left
on device)' in 53 files matching 'sn-devel-184'
  10 UNEXPECTED(failure):
samba.wbinfo_simple.check-secret.domain=SAMBA-TEST.wbinfo(nt4_member:local)
   5 UNEXPECTED(failure):
samba4.ldap.password_lockout.python(ad_dc_slowtests).__main__.PasswordTestsWithSleep.test_login_lockout_krb5(ad_dc_slowtests)
   4 UNEXPECTED(failure): samba3.smb2.notify.valid-req(nt4_dc)
   3 UNEXPECTED(failure):
samba4.ldap.notification.python(ad_dc_ntvfs).__main__.LDAPNotificationTest.test_simple_search(ad_dc_ntvfs)
   2 UNEXPECTED(failure): samba3.smb2.notify.mask(nt4_dc)
   2 *FAILED* tests/simple/60_recoverd_missing_ip.sh
   2 *FAILED* tests/simple/69_recovery_resurrect_deleted.sh
   2 UNEXPECTED(failure): samba3.raw.notify.mask(nt4_dc)
   2 *FAILED* tests/simple/27_ctdb_detach.sh

The metric of significance in this comparison is that I notice
something. I notice:

* password_lockout...PasswordTestsWithSleep fails 5 times in 18.04
  but only once in 14.04.

* 18_ctdb_reloadips.sh fails less with 18.04.

Everything else (and possibly these two) can be written off as noise.

Now I'm switching to looking at all our 18.04 bugs since March 1,
not just the ones that overlap with the final days of 14.04.

We had 17 wbinfo related failures:

  17 UNEXPECTED(failure):
samba.wbinfo_simple.check-secret.domain=SAMBA-TEST.wbinfo(nt4_member:local)

16 notify related:

   5 UNEXPECTED(failure): samba3.raw.notify.mask(nt4_dc)
   4 UNEXPECTED(failure): samba3.smb2.notify.valid-req(nt4_dc)
   3 UNEXPECTED(failure): samba3.smb2.notify.mask(nt4_dc)
   2 UNEXPECTED(failure): samba3.raw.notify.dir(nt4_dc)
   2 UNEXPECTED(failure): samba3.smb2.notify.dir(nt4_dc)

16 that had something to so with ctdb:

   5 *FAILED* tests/simple/27_ctdb_detach.sh
   4 *FAILED* tests/simple/69_recovery_resurrect_deleted.sh
   2 *FAILED* tests/simple/60_recoverd_missing_ip.sh
   2 *FAILED* tests/simple/18_ctdb_reloadips.sh
   1 *FAILED* tests/simple/32_ctdb_disable_enable.sh
   1 *FAILED* tests/cunit/run_event_001.sh
   1 *FAILED* tests/simple/91_version_check.sh

16 auth_log_winbind failures:

  10 UNEXPECTED(failure):
samba.tests.auth_log_winbind.samba.tests.auth_log_winbind.AuthLogTestsWinbind.test_wbinfo(ad_member:local)
   6 UNEXPECTED(failure):
samba.tests.auth_log_winbind.samba.tests.auth_log_winbind.AuthLogTestsWinbind.test_wbinfo_ntlmv1(ad_member:local)

Nine password lockout related:

   6 UNEXPECTED(failure):
samba4.ldap.password_lockout.python(ad_dc_slowtests).__main__.PasswordTestsWithSleep.test_login_lockout_krb5(ad_dc_slowtests)
   3 UNEXPECTED(failure):
samba4.ldap.password_lockout.python(ad_dc_slowtests).__main__.PasswordTestsWithSleep.test_pso_login_lockout_krb5(ad_dc_slowtests)

Five of these:

   5 UNEXPECTED(failure):
samba4.ldap.notification.python(ad_dc_ntvfs).__main__.LDAPNotificationTest.test_simple_search(ad_dc_ntvfs)

Five claiming to be about audit:

   2 UNEXPECTED(failure):
lib.audit_logging.audit_logging.test_audit_get_timestamp(none)
   1 UNEXPECTED(failure):
samba4.dsdb.samdb.ldb_modules.group_audit.test_audit_group_json(none)
   1 UNEXPECTED(failure):
samba4.dsdb.samdb.ldb_modules.group_audit.test_audit_group_json_no_event(none)
   1 UNEXPECTED(failure):
samba4.dsdb.samdb.ldb_modules.group_audit.test_audit_group_json_error(none)

Five RPC failures:

   1 UNEXPECTED(failure): samba4.rpc.backupkey with
seal.backupkey.server_wrap_encrypt_decrypt_wrong_key(ad_dc_default)
   1 UNEXPECTED(failure): samba4.rpc.altercontext on ncalrpc with
validate.altercontext(ad_dc_default:local)
   1 UNEXPECTED(failure): samba4.rpc.altercontext on ncalrpc with
bigendian.altercontext(ad_dc_default:local)
   1 UNEXPECTED(failure): samba4.rpc.altercontext on ncalrpc with
seal,padcheck.altercontext(ad_dc_default:local)
   1 UNEXPECTED(failure): samba4.rpc.backupkey with
seal.backupkey.server_wrap_decrypt_wrong_r2(ad_dc_default)

And six others:

   2 UNEXPECTED(failure):
samba4.drs.getnc_schema.python(schema_pair_dc).getnc_schema.SchemaReplicationTests.test_schema_linked_attributes(schema_pair_dc)
   2 UNEXPECTED(error):
samba4.drs.samba_tool_drs.python(vampire_dc).samba_tool_drs.SambaToolDrsTests.test_samba_tool_kcc(vampire_dc:local)
   1 UNEXPECTED(failure):
samba4.winbind.struct.lookup_name_sid(nt4_member:local)
   1 UNEXPECTED(failure):
samba4.ldap.rodc_rwdc.python(rodc).__main__.RodcRwdcCachedTests.test_login_lockout_krb5(rodc:local)

Last time I did this (2019-02-22) I found:
> So at the crudest level we have:
> 
> wbinfo:         11
> replica_sync:   10
> notify:          8
> samba_kcc:       3
> all others:      5

Now we have:

wbinfo:            17
notify:            16
ctdb:              16
auth_log_winbind:  16
password_lockout:   9
ldap.notification:  5
audit:              5
rpc:                5
others:             6

which looks worse if you don't take into account the periods of time.

The to 2019-02-22 numbers are looking at ~53 days or 212 test runs,
while today's numbers look at ~99 days or 396 tests. Converting to
percentage chance of failure per test:

wbinfo:             4.3
notify:             4.0
ctdb:               4.0
auth_log_winbind:   4.0
password_lockout:   2.3
ldap.notification:  1.3
audit:              1.3
rpc:                1.3
others:             1.5
in total:           24

That is, our flakey tests have been failing 24% of the time. In the
previous time period, they failed 17.5% of the time, so it is
unfortunately true that we made things worse.

cheers,
Douglas