A single tests.py, but still supporting make test in the autoconf build

[Andrew Bartlett]

On Mon, 2012-04-30 at 12:22 +0200, Michael Adam wrote:
> Andrew Bartlett wrote:
> >
> > Also, python tests get a samba. name already.
> > 
> > On Thu, 2012-04-26 at 14:33 +0200, Michael Adam wrote:
> > > What would be a good systematic naming?
> > > Is the "plugin_s4_dc" environent part of the pure source3 testsuite
> > > at all? Or is it rather s/th that is part of the s4-testsuite?
> > > 
> > > Maybe we should move to a common top level tests.py and have
> > > the top level and the source3 make test select an appropriate
> > > list of tests and environments? Just brainstorming here, not
> > > sure whether this is (easily) possible.
> > 
> > It is this final suggestion that I'm aiming for.  I have already blurred
> > the line between samba3 and samba4 tests, and I want to remove it
> > entirely.  Earlier, I restored the behaviour of selftest.pl to ignore
> > unknown environments, so we can (and do) declare tests in either
> > tests.py without issue.
> > 
> > I'm starting with the move of these particular tests as a place to
> > start, and because they are most critical to getting s3fs tested.  Even
> > moving these to a samba. prefix makes them more consistent with the
> > ptyon, but naturally I hope to finish with as consistent set of overall
> > test names as possible. 
> > 
> > Once I sort this particular area out, I hope to merge this all into one
> > tests.py (or multiple files, grouped around the type of test, rather
> > than the original script heritage).  We would then use skip and
> > knownfail rather than simply omitting to mention a test as the
> > controlling elements.  If the team is OK with this, it would essentially
> > change the s3 tests from the 'opt in' to 'opt out', and hopefully
> > increase our test coverage. 
> 
> Yes, that seems reasonable! It would basically mean skipping a couple
> of environments (the non-s3-envs) when running the source3-selftest.
> 
> The source3 "make test" can then still depend on the s3-autoconf
> build and call into the common selftest framework with a
> different set of paths and a filter on the tests listed by
> tests.py, possibly by skip patterns.

Essentially, yes.  

It's not actually the skip patterns, but the fact that the --target to
selftest.pl is samba3 not samba.  This causes selftest.pl to know about
a different set of environments that are possible, so so to skip tests
declared for environments that do not exist (including those that do not
exist due to lack of krb5 support at build time). 

Andrew Bartlett

-- 
Andrew Bartlett                                http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org