subunit name challenges: Attempting to run tests for plugin_s4_dc

[Andrew Bartlett]

On Thu, 2012-04-26 at 14:33 +0200, Michael Adam wrote:
> Hi Andrew,
> 
> generally, I like the patch-idea very much.
> 
> The patch misses the flapping-part (corresponding to knownfail
> and skip).
> 
> Regarding the nomenclature, I am not certain, as it is
> not completely consistent:
> 
> * the tests from source4/selftest/ get a "samba4" prefix
> * the tests from source3/selftest/ that are done with
>   plansmbtorturetestsuite have a "samba" prefix
> * the tests from source3/selftest/ that are done with
>   plantestsuite directly are done with a "samba3" prefix

Also, python tests get a samba. name already. 

> What would be a good systematic naming?
> Is the "plugin_s4_dc" environent part of the pure source3 testsuite
> at all? Or is it rather s/th that is part of the s4-testsuite?
> 
> Maybe we should move to a common top level tests.py and have
> the top level and the source3 make test select an appropriate
> list of tests and environments? Just brainstorming here, not
> sure whether this is (easily) possible.

It is this final suggestion that I'm aiming for.  I have already blurred
the line between samba3 and samba4 tests, and I want to remove it
entirely.  Earlier, I restored the behaviour of selftest.pl to ignore
unknown environments, so we can (and do) declare tests in either
tests.py without issue.

I'm starting with the move of these particular tests as a place to
start, and because they are most critical to getting s3fs tested.  Even
moving these to a samba. prefix makes them more consistent with the
ptyon, but naturally I hope to finish with as consistent set of overall
test names as possible. 

Once I sort this particular area out, I hope to merge this all into one
tests.py (or multiple files, grouped around the type of test, rather
than the original script heritage).  We would then use skip and
knownfail rather than simply omitting to mention a test as the
controlling elements.  If the team is OK with this, it would essentially
change the s3 tests from the 'opt in' to 'opt out', and hopefully
increase our test coverage. 

Thanks for looking at this with me!

Andrew Bartlett

-- 
Andrew Bartlett                                http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org