Test infrastructure improvements

[Andrew Bartlett]

On Sat, 2014-11-01 at 17:19 +0100, Jelmer Vernooij wrote:
> On Sat, Oct 25, 2014 at 03:06:48PM -0700, Jelmer Vernooij wrote:
> > Matthieu and I got together a couple of weeks ago to hack on the
> > Samba testsuite.
> > 
> > Our main goal has so far been on reducing the overall run time of the
> > testsuite, through various means. Matthieu has been pesking me about
> > running tests in parallel for a long time.
> > 
> > I've put the todo-list we compiled on the wiki:
> > https://wiki.samba.org/index.php/Test_Infrastructure_Sprint
> > 
> > What we have at the moment:
> > 
> > * drop the samba3/samba4 distinction?
> I think this is a much larger change, since it ties in with other code changes
> to e.g. remove the distinction between smbtorture3 and smbtorture4.
> 
> Perhaps we should hold off on this as part of the test infrastructure
> improvements, though I agree it would be a nice thing to do in general.

At one level, the samba, samba3. and samba4. prefixes have become quite
meaningless.  That is, they refer mostly to the file the tests are
declared in, that is selftest/tests.py, source3/selftest/tests.py and
source4/selftest/tests.py.  

The different files used to matter when we had two different build
systems, because we ran one or the other, and then later ran either both
or only the source3 files. 

However, in each of these files, tests are run against arbitrary
environments and both call smbtorture for much of the work.  That means
the name isn't really a reflection of if the test is a 'samba3' or
'samba4' test any more.  The primary difference that I'm aware of is
that the --target={samba3,samba4} can change some behaviours in the
test.

We should work to ensure we run all tests against all reasonable
combinations of environments, and we should rename the environments to
have more reasonable names, so you can work more clearly out what your
failing test actually is.  Moving away from 3/4 naming would be a plus,
provided we get a clear alternative. 

We should also try and move our multitude of tests.py files to either
one file, or split it out in some reasonable way, as currently the
allocation is essentially historic. 

As a start, perhaps all the python tests should move up to
selftest/tests.py?

Andrew Bartlett

-- 
Andrew Bartlett
http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba