knownfail or skip for flakey tests?
abartlet at samba.org
Fri Dec 9 04:50:47 MST 2011
On Tue, 2011-12-06 at 00:51 +0100, Jelmer Vernooij wrote:
> Am 14/10/11 08:14, schrieb Stefan (metze) Metzmacher:
> > Hi,
> >>>> after discussing with Metze, I marked the
> >>>> samba4.drs.delete_object.python test knownfail.
> >>>> It seems to be flakey:
> >>>> http://git.samba.org/autobuild.flakey/2011-10-12-1328/samba4.stdout
> >>> If the test is flaky, it should probably be skipped. having a test in
> >>> knownfail means that it will be reported as a failure if the test
> >>> actually succeeds.
> >> Jelmer,
> >> I know this is meant to be the case, but I'm pretty sure it isn't how it
> >> is working for now. We have been putting flaky tests in knownfail for
> >> quite some time now, which allows us to tell the difference between
> >> 'fails' and 'segfaults'.
> > Sadly currently knownfail just means that we ignore failures,
> > but we don't turn unexpected success into an error.
> > We used to do that a few years ago, before we got
> > selftest/format-subunit and
> > selftest/filter-subunit as external processes.
> > I think we should try to fix that and add an additional
> > handling for flakey tests. We could then maintain
> > flakey-failures and flakey-errors files and also ignore errors
> > if needed instead of only failures.
> This has now been fixed in master. Tests listed in selftest/knownfail
> that actually succeed will now once again trigger a uxsuccess
> (unexpected success), which is considered an error.
> I've moved those tests in knownfail that had a comment saying they were
> flapping to selftest/flapping. Several tests could also be removed from
> knownfail because they are no longer failing (ie because remote calls
> have been implemented, or bugs fixed). If you run into any more tests
> that are flapping, please add them there as well.
Thank you very much for doing this. It is great to have these lists
sorted out, and I have been able to remove some from skip and knownfail
with only small changes, which improves things even more.
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
More information about the samba-technical