Test infrastructure improvements

[Jelmer Vernooij]

On Mon, Nov 03, 2014 at 10:13:39AM +1300, Andrew Bartlett wrote:
> On Sat, 2014-11-01 at 17:19 +0100, Jelmer Vernooij wrote:
> > On Sat, Oct 25, 2014 at 03:06:48PM -0700, Jelmer Vernooij wrote:
> > > Matthieu and I got together a couple of weeks ago to hack on the
> > > Samba testsuite.
> > > 
> > > Our main goal has so far been on reducing the overall run time of the
> > > testsuite, through various means. Matthieu has been pesking me about
> > > running tests in parallel for a long time.
> > > 
> > > I've put the todo-list we compiled on the wiki:
> > > https://wiki.samba.org/index.php/Test_Infrastructure_Sprint
> > > 
> > > What we have at the moment:
> > > 
> > > * drop the samba3/samba4 distinction?
> > I think this is a much larger change, since it ties in with other code changes
> > to e.g. remove the distinction between smbtorture3 and smbtorture4.
> > 
> > Perhaps we should hold off on this as part of the test infrastructure
> > improvements, though I agree it would be a nice thing to do in general.
> 
> At one level, the samba, samba3. and samba4. prefixes have become quite
> meaningless.  That is, they refer mostly to the file the tests are
> declared in, that is selftest/tests.py, source3/selftest/tests.py and
> source4/selftest/tests.py.  
They also refer to the version of the tools that is used to run the tests. E.g.
the samba3 tests use the source3 smbtorture, the samba4 ones use the source4
smbtorture.

If we're going to remove the distinction in the names, then we should also
reconcile the underlying code.

> The different files used to matter when we had two different build
> systems, because we ran one or the other, and then later ran either both
> or only the source3 files. 
I think they still somewhat matter for the reasons mentioned above.

> However, in each of these files, tests are run against arbitrary
> environments and both call smbtorture for much of the work.  That means
> the name isn't really a reflection of if the test is a 'samba3' or
> 'samba4' test any more.  The primary difference that I'm aware of is
> that the --target={samba3,samba4} can change some behaviours in the
> test.
The --target parameter is supposed to match the environment (e.g. "samba3" for
"s3dc"), not the origin of the test itself (which is what the samba3/samba4 prefix
represents).

> We should also try and move our multitude of tests.py files to either
> one file, or split it out in some reasonable way, as currently the
> allocation is essentially historic. 
I think once we've merged a specific bit of code, that test can be moved to
the top-level tests.py.

> As a start, perhaps all the python tests should move up to
> selftest/tests.py?
That's a good idea.

Cheers,

Jelmer