Flapping tests during the 4.8rc window
Douglas Bagnall
douglas.bagnall at catalyst.net.nz
Thu Mar 15 03:06:45 UTC 2018
This is about the tests that have flapped in the 6-hourly autobuilds
of the master branch on sn-devel, as revealed by emails to the
samba-cvs list, during the period from 2018-01-11 until about now.
The main reason for choosing this period is immediately before that
there was a lot of chaos caused by three flapping tests that have now
been safely swept under the carpet.
According to the flakey.log files, failures fell here:
54 samba: [test] failed
2 ctdb: [test] failed
1 tdb: [test] failed
Looking at samba first:
7 UNEXPECTED(error): samba.tests.blackbox.smbcontrol.samba.tests.blackbox.smbcontrol.SmbcontrolBlockboxTests.test_ping(ad_dc_ntvfs:local)
7 UNEXPECTED(failure): samba.wbinfo_simple.(nt4_member:local).--check-secret --domain=SAMBA-TEST.wbinfo(nt4_member:local)
5 UNEXPECTED(failure): samba4.ldap.possibleInferiors.python(ad_dc_ntvfs).objectClass.lostAndFound(ad_dc_ntvfs)
5 UNEXPECTED(failure): samba4.blackbox.trust_ntlm.Test05 rpcclient getusername with UNKNOWNDOMAIN\autobuild%localnt4member3pass(nt4_member:local)
4 UNEXPECTED(error): samba.tests.blackbox.smbcontrol.samba.tests.blackbox.smbcontrol.SmbcontrolBlockboxTests.test_ping(fl2003dc:local)
3 UNEXPECTED(failure): samba.tests.samba_tool.user_wdigest.samba.tests.samba_tool.user_wdigest.UserCmdWdigestTestCase.test_Wdigest01(ad_dc_ntvfs:local)
2 UNEXPECTED(error): samba4.ldap_schema.python(ad_dc_ntvfs).__main__.SchemaTests.test_generated_mAPIID(ad_dc_ntvfs)
2 UNEXPECTED(failure): samba.wbinfo_sids2xids.(ad_member:local).sids2xids(ad_member:local)
2 UNEXPECTED(failure): samba.tests.samba_tool.visualize_drs.samba.tests.samba_tool.visualize_drs.SambaToolVisualizeDrsTest.test_reps_remote(vampire_dc)
2 UNEXPECTED(failure): samba.wbinfo_simple.check-secret.domain=SAMBA-TEST.wbinfo(nt4_member:local)
1 UNEXPECTED(error): samba4.drs.getncchanges.python(promoted_dc).getncchanges.DrsReplicaSyncIntegrityTestCase.test_repl_integrity_obj_reanimation(promoted_dc)
1 UNEXPECTED(error): samba4.drs.getncchanges.python(promoted_dc).getncchanges.DrsReplicaSyncIntegrityTestCase.test_repl_integrity(promoted_dc)
1 UNEXPECTED(error): samba4.drs.replica_sync.python(promoted_dc).replica_sync.DrsReplicaSyncTestCase.test_ReplConflictsRemoteWin(promoted_dc:local)
1 UNEXPECTED(error): samba4.drs.repl_rodc.python(ad_dc_ntvfs).repl_rodc.DrsRodcTestCase.test_msDSRevealedUsers(ad_dc_ntvfs)
1 UNEXPECTED(failure): samba.tests.blackbox.samba_dnsupdate.samba.tests.blackbox.samba_dnsupdate.SambaDnsUpdateTests.test_samba_dnsupate_set_ip(chgdcpass:local)
1 UNEXPECTED(failure): samba3.smb2.notify.mask(ad_dc)
1 UNEXPECTED(error): samba4.ldap_schema.python(ad_dc_ntvfs).__main__.SchemaTests.test_subClassOf(ad_dc_ntvfs)
1 UNEXPECTED(failure): samba4.ldap.possibleInferiors.python(ad_dc_ntvfs).objectClass.dfsConfiguration(ad_dc_ntvfs)
1 UNEXPECTED(error): samba.tests.join.python(ad_dc_ntvfs).samba.tests.join.JoinTestCase.test_join_makes_records(ad_dc_ntvfs)
1 UNEXPECTED(failure): samba4.blackbox.kpasswd(ad_dc_ntvfs:local).kinit with user password(ad_dc_ntvfs:local)
1 UNEXPECTED(error): samba4.drs.replica_sync.python(vampire_dc).replica_sync.DrsReplicaSyncTestCase.test_ReplConflictsRemoteWin_with_child(vampire_dc:local)
1 UNEXPECTED(error): samba.tests.blackbox.smbcontrol.samba.tests.blackbox.smbcontrol.SmbcontrolBlockboxTests.test_ping(fl2008r2dc:local)
1 UNEXPECTED(failure): samba3.blackbox.net_tdb.Looking for open file in locking.tdb(simpleserver:local)
Of these, at least the smbcontrol and user_wdigest problems have been
fixed. Five of the failures were caused by timeout problems last
weekend, so we can ignore those, leaving:
7 UNEXPECTED(failure): samba.wbinfo_simple.(nt4_member:local).--check-secret --domain=SAMBA-TEST.wbinfo(nt4_member:local)
5 UNEXPECTED(failure): samba4.ldap.possibleInferiors.python(ad_dc_ntvfs).objectClass.lostAndFound(ad_dc_ntvfs)
5 UNEXPECTED(failure): samba4.blackbox.trust_ntlm.Test05 rpcclient getusername with UNKNOWNDOMAIN\autobuild%localnt4member3pass(nt4_member:local)
2 UNEXPECTED(error): samba4.ldap_schema.python(ad_dc_ntvfs).__main__.SchemaTests.test_generated_mAPIID(ad_dc_ntvfs)
2 UNEXPECTED(failure): samba.wbinfo_sids2xids.(ad_member:local).sids2xids(ad_member:local)
2 UNEXPECTED(failure): samba.wbinfo_simple.check-secret.domain=SAMBA-TEST.wbinfo(nt4_member:local)
1 UNEXPECTED(error): samba4.drs.getncchanges.python(promoted_dc).getncchanges.DrsReplicaSyncIntegrityTestCase.test_repl_integrity_obj_reanimation(promoted_dc)
1 UNEXPECTED(error): samba4.drs.replica_sync.python(promoted_dc).replica_sync.DrsReplicaSyncTestCase.test_ReplConflictsRemoteWin(promoted_dc:local)
1 UNEXPECTED(failure): samba.tests.blackbox.samba_dnsupdate.samba.tests.blackbox.samba_dnsupdate.SambaDnsUpdateTests.test_samba_dnsupate_set_ip(chgdcpass:local)
1 UNEXPECTED(failure): samba3.smb2.notify.mask(ad_dc)
1 UNEXPECTED(error): samba4.ldap_schema.python(ad_dc_ntvfs).__main__.SchemaTests.test_subClassOf(ad_dc_ntvfs)
1 UNEXPECTED(failure): samba4.ldap.possibleInferiors.python(ad_dc_ntvfs).objectClass.dfsConfiguration(ad_dc_ntvfs)
1 UNEXPECTED(failure): samba4.blackbox.kpasswd(ad_dc_ntvfs:local).kinit with user password(ad_dc_ntvfs:local)
1 UNEXPECTED(error): samba4.drs.replica_sync.python(vampire_dc).replica_sync.DrsReplicaSyncTestCase.test_ReplConflictsRemoteWin_with_child(vampire_dc:local)
1 UNEXPECTED(failure): samba3.blackbox.net_tdb.Looking for open file in locking.tdb(simpleserver:local)
Of these,
blackbox.samba_dnsupdate.SambaDnsUpdateTests.test_samba_dnsupate_set_ip(chgdcpass:local),
replica_sync.DrsReplicaSyncTestCase.test_ReplConflictsRemoteWin_with_child(vampire_dc:local), and
samba4.ldap.possibleInferiors.python(ad_dc_ntvfs).objectClass.dfsConfiguration(ad_dc_ntvfs)
have been troubling us for a while (occurring in at least 5 months of the last year).
The ctdb failures were both in tests/eventd/eventd_024.sh
https://git.samba.org/autobuild.flakey.sn-devel-144/2018-01-31-0016/ctdb.stdout
https://git.samba.org/autobuild.flakey.sn-devel-144/2018-03-14-0008/ctdb.stdout
and the tdb failure was
https://git.samba.org/autobuild.flakey.sn-devel-144/2018-02-28-1205/tdb.stdout
tdb1-run-allrecord-traverse-deadlock failed:
........................marklock-deadlock-fcntl.tdb tests done
..................do_allrecord_lock: traverse should fail
......ret should succeed
Finally, here's the histogram of failure, showing that February was
our least flappy month on record!
2015-12 18 ##################
2016-01 36 ####################################
2016-02 35 ###################################
2016-03 47 ###############################################
2016-04 49 #################################################
2016-05 55 #######################################################
2016-06 58 ##########################################################
2016-07 53 #####################################################
2016-08 50 ##################################################
2016-09 24 ########################
2016-10 22 ######################
2016-11 23 #######################
2016-12 22 ######################
2017-01 44 ############################################
2017-02 29 #############################
2017-03 22 ######################
2017-04 35 ###################################
2017-05 45 #############################################
2017-06 64 ################################################################
2017-07 26 ##########################
2017-08 21 #####################
2017-09 27 ###########################
2017-10 38 ######################################
2017-11 25 #########################
2017-12 50 ##################################################
2018-01 35 ###################################
2018-02 17 #################
2018-03 25 #########################
It would be nicer if we'd got there with [ab]using selftest/flapping.d
but never mind.
Douglas
More information about the samba-technical
mailing list