knownfail or skip for flakey tests?

[Andrew Bartlett]

On Thu, 2011-10-13 at 16:34 +0200, Jelmer Vernooij wrote:
> Hi Michael,
> 
> On 10/13/2011 12:34 AM, Michael Adam wrote:
> > Michael Adam wrote:
> >> The branch, master has been updated
> >> via 7e9acf0 selftest: mark samba4.drs.delete_object.python knownfail
> >> ...
> >> from c6481f4 samba.upgrade: Use list comprehension.
> >>
> >> http://gitweb.samba.org/?p=samba.git;a=shortlog;h=master
> >>
> >>
> >> - Log -----------------------------------------------------------------
> >> commit 7e9acf06c5b4c8f77f092f35483b02eb9952b97e
> >> Author: Michael Adam <obnox at samba.org>
> >> Date: Wed Oct 12 22:11:26 2011 +0200
> >>
> >> selftest: mark samba4.drs.delete_object.python knownfail
> >>
> >> this currently prevents autobuild from succeeding intermittently
> >>
> >> Autobuild-User: Michael Adam <obnox at samba.org>
> >> Autobuild-Date: Thu Oct 13 00:22:18 CEST 2011 on sn-devel-104
> >
> > Hi folks,
> >
> > after discussing with Metze, I marked the
> > samba4.drs.delete_object.python test knownfail.
> > It seems to be flakey:
> > http://git.samba.org/autobuild.flakey/2011-10-12-1328/samba4.stdout
> If the test is flaky, it should probably be skipped. having a test in
> knownfail means that it will be reported as a failure if the test
> actually succeeds.

Jelmer,

I know this is meant to be the case, but I'm pretty sure it isn't how it
is working for now.  We have been putting flaky tests in knownfail for
quite some time now, which allows us to tell the difference between
'fails' and 'segfaults'.

Andrew Bartlett
-- 
Andrew Bartlett                                http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org