flapping tests summary for November 2017

Fri Dec 1 03:21:20 UTC 2017

The hottest new trend is for flapping tests to have very long names
with internal redundancy (though the succinct "samba3.unix.whoami"
is always in style):

$ ./parse-email/parse-autobuild-email --file-regex sn-devel-144  --since 2017-11-01 --filter-line=': ([\w.]+)'
found 24 lines matching '^UNEXPECTED' in 26 files matching 'sn-devel-144'
   5 samba4.drs.repl_move.python
   3 samba3.unix.whoami
   3 samba4.ldap_schema.python
   2 samba4.drs.replica_sync.python
   2 samba4.drs.getncchanges.python
   1 samba.tests.samba_tool.user_wdigest.samba.tests.samba_tool.user_wdigest.UserCmdWdigestTestCase.test_Wdigest01
   1 samba.tests.dsdb_schema_attributes.samba.tests.dsdb_schema_attributes.SchemaAttributesTestCase.test_modify_at_indexlist
   1 samba4.drs.samba_tool_drs.python
   1 samba.ntlm_auth.diagnostics
   1 samba3.smb2.notify.dir
   1 samba.tests.blackbox.samba_dnsupdate.samba.tests.blackbox.samba_dnsupdate.SambaDnsUpdateTests.test_samba_dnsupate_set_ip
   1 samba4.ldap.possibleInferiors.python
   1 samba.tests.dsdb_schema_attributes.samba.tests.dsdb_schema_attributes.SchemaAttributesTestCase.test_modify_at_attributes
   1 idmap.rfc2307.Testing

Those are the testsuite names. When we look at the exact tests, we see
only one flapping test per suite, except for samba4.ldap_schema.python
which has two:

$ ./parse-email/parse-autobuild-email --file-regex sn-devel-144  --since 2017-11-01 found 24 lines matching '^UNEXPECTED' in 26 files matching 'sn-devel-144'
   5 UNEXPECTED(failure): samba4.drs.repl_move.python(vampire_dc).repl_move.DrsMoveBetweenTreeOfObjectTestCase.test_ReplicateAddInConflictOU2(vampire_dc)
   3 UNEXPECTED(failure): samba3.unix.whoami machine account.whoami(nt4_member:local)
   2 UNEXPECTED(error): samba4.drs.replica_sync.python(promoted_dc).replica_sync.DrsReplicaSyncTestCase.test_ReplConflictsRenamedVsNewRemoteWin(promoted_dc:local)
   2 UNEXPECTED(failure): samba4.ldap_schema.python(ad_dc_ntvfs).__main__.SchemaTests.test_generated_linkID(ad_dc_ntvfs)
   2 UNEXPECTED(failure): samba4.drs.getncchanges.python(vampire_dc).getncchanges.DrsReplicaSyncIntegrityTestCase.test_repl_integrity_cross_partition_links(vampire_dc)
   1 UNEXPECTED(failure): samba.ntlm_auth.diagnostics(s4member:local).ntlm_auth(s4member:local)
   1 UNEXPECTED(failure): samba.tests.blackbox.samba_dnsupdate.samba.tests.blackbox.samba_dnsupdate.SambaDnsUpdateTests.test_samba_dnsupate_set_ip(chgdcpass:local)
   1 UNEXPECTED(error): samba.tests.dsdb_schema_attributes.samba.tests.dsdb_schema_attributes.SchemaAttributesTestCase.test_modify_at_indexlist(ad_dc_ntvfs:local)
   1 UNEXPECTED(failure): samba.tests.samba_tool.user_wdigest.samba.tests.samba_tool.user_wdigest.UserCmdWdigestTestCase.test_Wdigest01(ad_dc_ntvfs:local)
   1 UNEXPECTED(error): samba4.drs.samba_tool_drs.python(promoted_dc).samba_tool_drs.SambaToolDrsTests.test_samba_tool_kcc(promoted_dc:local)
   1 UNEXPECTED(error): samba4.ldap_schema.python(ad_dc_ntvfs).__main__.SchemaTests.test_subClassOf(ad_dc_ntvfs)
   1 UNEXPECTED(failure): samba4.ldap.possibleInferiors.python(ad_dc_ntvfs).objectClass.dfsConfiguration(ad_dc_ntvfs)
   1 UNEXPECTED(failure): idmap.rfc2307.Testing for expected group memberships(ad_member_rfc2307)
   1 UNEXPECTED(error): samba.tests.dsdb_schema_attributes.samba.tests.dsdb_schema_attributes.SchemaAttributesTestCase.test_modify_at_attributes(ad_dc_ntvfs:local)
   1 UNEXPECTED(failure): samba3.smb2.notify.dir(nt4_dc)

The monthly histogram shows nothing remarkable in the last three
months:

2015-11   5 #####
2015-12  18 ##################
2016-01  36 ####################################
2016-02  35 ###################################
2016-03  47 ###############################################
2016-04  49 #################################################
2016-05  55 #######################################################
2016-06  58 ##########################################################
2016-07  53 #####################################################
2016-08  50 ##################################################
2016-09  24 ########################
2016-10  22 ######################
2016-11  23 #######################
2016-12  22 ######################
2017-01  44 ############################################
2017-02  29 #############################
2017-03  22 ######################
2017-04  35 ###################################
2017-05  45 #############################################
2017-06  64 ################################################################
2017-07  26 ##########################
2017-08  21 #####################
2017-09  27 ###########################
2017-10  38 ######################################
2017-11  25 #########################
2017-12   1 #

The new feature this time is flakey.log parsing. The above results
assume the samba tests to blame and look only in the samba.stdout
file. But there are other tests running in parallel and sometimes they
flap. For November we have:

$ ./parse-email/parse-autobuild-email   --flakey-log --file-regex sn-devel-144 --since 2017-11-01 --filter-line '^(\w+: \[\w+\] failed)'
found 52 lines matching "\\w+: \\[\\w+\\] failed '.+' with status" in 26 files matching 'flakey.log$'
  48 samba: [test] failed
   4 ctdb: [test] failed

and from the beginning of time (~2015):

$ ./parse-email/parse-autobuild-email   --flakey-log --file-regex sn-devel-144  --filter-line '^(\w+: \[\w+\] failed)'
found 2529 lines matching "\\w+: \\[\\w+\\] failed '.+' with status" in 3264 files matching 'flakey.log$'
2129 samba: [test] failed
 211 ctdb: [test] failed
 119 samba: [make] failed
  31 samba: [install] failed
  24 samba: [configure] failed
   4 tevent: [make] failed
   4 talloc: [make] failed
   3 ldb: [make] failed
   2 tdb: [test] failed
   1 tdb: [make] failed
   1 ntdb: [test] failed

It looks like test flakiness is roughly proportional to developer
effort.

As always these are based on emails sent to the samba-cvs list[1]
by a system that automatically tests the canonical git repository. 

[1] https://lists.samba.org/archive/samba-cvs/

Douglas