selftest: mark the samba4.blackbox.dbcheck test as flapping.

Andrew Bartlett abartlet at samba.org
Fri Mar 20 00:33:45 MDT 2015


On Thu, 2015-03-19 at 20:44 +0100, Michael Adam wrote:
> The branch, master has been updated
>        via  77ba781 selftest: mark the samba4.blackbox.dbcheck test as flapping.
>        via  04d6ef8 selftest: mark the samba4.blackbox.samba_tool_demote test flakey.
>        via  90125e6 s3:torture: add samba3.smbtorture_s3.LOCAL-MESSAGING-FDPASS2b test.
>        via  a4f7bc5 s3:torture: add samba3.smbtorture_s3.LOCAL-MESSAGING-FDPASS2a test.
>        via  bf14864 s3:torture: prepare the FDPASS2 test to be run with variable payload sizes.
>        via  57e3eba fix failing fd passing message by passing needed buffer size
>        via  7551a4e vfs_ceph: remove cephwrap_init_stat_ex_from_stat()
>        via  16956d4 vfs_ceph: fix ntimes_fn callback
>       from  5f01bb1 build: Add talloc and samba-debug dep for gensec_external module
> 
> https://git.samba.org/?p=samba.git;a=shortlog;h=master
> 
> 
> - Log -----------------------------------------------------------------
> commit 77ba781c7ad3f71e1641962f4716165cab0ad20b
> Author: Michael Adam <obnox at samba.org>
> Date:   Thu Mar 19 16:59:42 2015 +0100
> 
>     selftest: mark the samba4.blackbox.dbcheck test as flapping.
>     
>     This causes frequent autobuild failures recently as follows:
>     
>     [1730(16550)/1735 at 2h7m43s] samba4.blackbox.dbcheck(ad_dc_ntvfs)
>     UNEXPECTED(failure): samba4.blackbox.dbcheck(ad_dc_ntvfs).dbcheck(ad_dc_ntvfs:local)
>     REASON: Exception: Exception: Checking 6799 objects
>     ERROR: wrong dn[CN=DrsDelObjUser_1426756150,CN=Deleted Objects,DC=samba,DC=example,DC=com] name='DrsDelObjUser_1426756150
>     DEL:77ec48cb-06cc-4157-a565-770720c30929' new_dn[CN=DrsDelObjUser_1426756150
>     
>     FAILED (1 failures, 0 errors and 0 unexpected successes in 0 testsuites)
>     
>     Has been seen flapping since quite some time:
>     
>     https://git.samba.org/autobuild.flakey/2012-06-22-0340/samba.stdout
>     https://git.samba.org/autobuild.flakey/2012-06-25-2339/samba.stdout
>     https://git.samba.org/autobuild.flakey/2012-07-02-1741/samba.stdout
>     https://git.samba.org/autobuild.flakey/2012-07-31-1344/samba.stdout
>     https://git.samba.org/autobuild.flakey/2012-09-24-1331/samba.stdout
>     https://git.samba.org/autobuild.flakey/2012-11-21-0933/samba.stdout
>     https://git.samba.org/autobuild.flakey/2012-12-02-0140/samba.stdout
>     https://git.samba.org/autobuild.flakey/2012-12-11-1933/samba.stdout
>     ...
>     https://git.samba.org/autobuild.flakey/2014-08-20-1715/samba.stdout
>     https://git.samba.org/autobuild.flakey/2014-09-29-0222/samba.stdout
>     https://git.samba.org/autobuild.flakey/2014-12-13-1119/samba.stdout
>     https://git.samba.org/autobuild.flakey/2015-03-19-0529/samba.stdout
>     https://git.samba.org/autobuild.flakey/2015-03-19-0828/samba.stdout
>     https://git.samba.org/autobuild.flakey/2015-03-19-1128/samba.stdout
>     
>     Possibly related to the failure of tombstone_ranimation,
>     drs.delete_object, and blackbox.samba_tool_demote.
>     
>     Signed-off-by: Michael Adam <obnox at samba.org>
>     Reviewed-by: Guenther Deschner <gd at samba.org>
>     
>     Autobuild-User(master): Michael Adam <obnox at samba.org>
>     Autobuild-Date(master): Thu Mar 19 20:43:32 CET 2015 on sn-devel-104

My concern with this change is that now a change that permanently breaks
this test won't ever be noticed, because we have no way to detect that
this test has gone from flapping (once every 10 days, eg 40 autobuilds
by your stats) to totally busted.

While trying to chase the tombstone_reanimation test certainly I've seen
a lot of flapping tests all over the code, which made looking for that
failure very difficult.  So, I'm not saying the current situation is
ideal, but I am concerned.  

The problem is particularly difficult given our limited resources -
limited resources to fix flapping tests, and limited resources to fix
regressions introduced and hidden behind flapping entries.

I would appreciate your further thoughts on how to address this further,
and to find a way we can qualify getting tests back of the flapping
list. 

Thanks,

Andrew Bartlett

-- 
Andrew Bartlett
http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba






More information about the samba-technical mailing list