knownfail or skip for flakey tests?

[Stefan (metze) Metzmacher]

Hi,

>>> after discussing with Metze, I marked the
>>> samba4.drs.delete_object.python test knownfail.
>>> It seems to be flakey:
>>> http://git.samba.org/autobuild.flakey/2011-10-12-1328/samba4.stdout
>> If the test is flaky, it should probably be skipped. having a test in
>> knownfail means that it will be reported as a failure if the test
>> actually succeeds.
> 
> Jelmer,
> 
> I know this is meant to be the case, but I'm pretty sure it isn't how it
> is working for now.  We have been putting flaky tests in knownfail for
> quite some time now, which allows us to tell the difference between
> 'fails' and 'segfaults'.

Sadly currently knownfail just means that we ignore failures,
but we don't turn unexpected success into an error.
We used to do that a few years ago, before we got
selftest/format-subunit and
selftest/filter-subunit as external processes.

I think we should try to fix that and add an additional
handling for flakey tests. We could then maintain
flakey-failures and flakey-errors files and also ignore errors
if needed instead of only failures.

metze


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 262 bytes
Desc: OpenPGP digital signature
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20111014/ee960e6a/attachment.pgp>