sn-devel flapping tests summary since 2017-02-10

[Douglas Bagnall]

This is the latest in an occasional series consisting of me running a
script[1] that counts intermittent failing tests, as indicated by all the
emails sent to the samba-cvs[2] list by the test robot that have
reached me.

In this case the date range has been selected to miss the masktest
failures that were discussed here and fixed in mid-February.

The tests that flapped more than once are:

$ ./parse-email/parse-autobuild-email --file-regex sn-devel-144 --since
2017-02-10 --filter-line=': ([\w.]+)'
found 54 lines matching '^UNEXPECTED' in 87 files matching 'sn-devel-144'
  19 samba4.drs.replica_sync.python
   6
samba.tests.dcerpc.dnsserver.samba.tests.dcerpc.dnsserver.DnsserverTests.test_accept_valid_commands
   3 samba4.drs.repl_move.python
   3
samba.tests.dcerpc.dnsserver.samba.tests.dcerpc.dnsserver.DnsserverTests.test_add_similar_value
   3 samba4.blackbox.masktest.masktest
   2 samba3.raw.notify.mask
   2 samba3.blackbox.smbclient_s3.crypt
   2 samba3.blackbox.smbclient_s3.sign
   2
samba.tests.blackbox.samba_dnsupdate.samba.tests.blackbox.samba_dnsupdate.SambaDnsUpdateTests.test_samba_dnsupate_set_ip

or in more detail:

$ ./parse-email/parse-autobuild-email --file-regex sn-devel-144 --since
2017-02-10
found 54 lines matching '^UNEXPECTED' in 87 files matching 'sn-devel-144'
   7 UNEXPECTED(failure):
samba4.drs.replica_sync.python(promoted_dc).replica_sync.DrsReplicaSyncTestCase.test_ReplRenames(promoted_dc:local)
   4 UNEXPECTED(failure):
samba.tests.dcerpc.dnsserver.samba.tests.dcerpc.dnsserver.DnsserverTests.test_accept_valid_commands(ad_dc)
   4 UNEXPECTED(error):
samba4.drs.replica_sync.python(vampire_dc).replica_sync.DrsReplicaSyncTestCase.test_ReplConflictsRemoteWin(vampire_dc:local)
   3 UNEXPECTED(error):
samba4.drs.repl_move.python(promoted_dc).repl_move.DrsMoveBetweenTreeOfObjectTestCase.test_ReplicateAddInConflictOU2(promoted_dc)
   3 UNEXPECTED(failure): samba4.blackbox.masktest.masktest(ad_dc_ntvfs)
   3 UNEXPECTED(error):
samba4.drs.replica_sync.python(vampire_dc).replica_sync.DrsReplicaSyncTestCase.test_ReplConflictsRemoteWin_with_child(vampire_dc:local)
   2 UNEXPECTED(error):
samba4.drs.replica_sync.python(promoted_dc).replica_sync.DrsReplicaSyncTestCase.test_ReplConflictsRemoteWin_with_child(promoted_dc:local)
   2 UNEXPECTED(failure):
samba.tests.dcerpc.dnsserver.samba.tests.dcerpc.dnsserver.DnsserverTests.test_accept_valid_commands(ad_dc_ntvfs)
   2 UNEXPECTED(failure): samba3.blackbox.smbclient_s3.sign (nt4_member)
member creds.follow symlinks = no(nt4_member)
   2 UNEXPECTED(failure):
samba.tests.blackbox.samba_dnsupdate.samba.tests.blackbox.samba_dnsupdate.SambaDnsUpdateTests.test_samba_dnsupate_set_ip(chgdcpass:local)
   2 UNEXPECTED(failure):
samba.tests.dcerpc.dnsserver.samba.tests.dcerpc.dnsserver.DnsserverTests.test_add_similar_value(ad_dc_ntvfs)
   2 UNEXPECTED(failure): samba3.raw.notify.mask(nt4_dc)
   2 UNEXPECTED(failure): samba3.blackbox.smbclient_s3.crypt
(nt4_dc).follow symlinks = no(nt4_dc)


Finally, here is a histogram to show how we've been going over time
(ignoring the abandonware known as "sn-devel-104"):

$ ./parse-email/parse-autobuild-email  --file-regex '(sn-devel-144)'
--histogram
2015-11   5 #####
2015-12  18 ##################
2016-01  36 ####################################
2016-02  35 ###################################
2016-03  47 ###############################################
2016-04  49 #################################################
2016-05  55 #######################################################
2016-06  58 ##########################################################
2016-07  53 #####################################################
2016-08  50 ##################################################
2016-09  24 ########################
2016-10  22 ######################
2016-11  23 #######################
2016-12  22 ######################
2017-01  44 ############################################
2017-02  29 #############################
2017-03  22 ######################
2017-04  18 ##################


We are getting better! More tests and fewer flaps. Luckily April has
been such a short month.

Douglas



[1]
http://git.catalyst.net.nz/gitweb?p=samba-cloud-autobuild.git;a=blob;f=parse-email/parse-autobuild-email
[2] https://lists.samba.org/archive/samba-cvs/