Test infrastructure improvements

[Andrew Bartlett]

On Sun, 2014-11-02 at 23:16 +0100, Jelmer Vernooij wrote:
> On Mon, Nov 03, 2014 at 10:13:39AM +1300, Andrew Bartlett wrote:
> > On Sat, 2014-11-01 at 17:19 +0100, Jelmer Vernooij wrote:
> > > On Sat, Oct 25, 2014 at 03:06:48PM -0700, Jelmer Vernooij wrote:
> > > > Matthieu and I got together a couple of weeks ago to hack on the
> > > > Samba testsuite.
> > > > 
> > > > Our main goal has so far been on reducing the overall run time of the
> > > > testsuite, through various means. Matthieu has been pesking me about
> > > > running tests in parallel for a long time.
> > > > 
> > > > I've put the todo-list we compiled on the wiki:
> > > > https://wiki.samba.org/index.php/Test_Infrastructure_Sprint
> > > > 
> > > > What we have at the moment:
> > > > 
> > > > * drop the samba3/samba4 distinction?
> > > I think this is a much larger change, since it ties in with other code changes
> > > to e.g. remove the distinction between smbtorture3 and smbtorture4.
> > > 
> > > Perhaps we should hold off on this as part of the test infrastructure
> > > improvements, though I agree it would be a nice thing to do in general.
> > 
> > At one level, the samba, samba3. and samba4. prefixes have become quite
> > meaningless.  That is, they refer mostly to the file the tests are
> > declared in, that is selftest/tests.py, source3/selftest/tests.py and
> > source4/selftest/tests.py.  
> They also refer to the version of the tools that is used to run the tests. E.g.
> the samba3 tests use the source3 smbtorture, the samba4 ones use the source4
> smbtorture.

No, this isn't the case.  Most of the tests in source3/selftest/tests.py
use smbtorture, as built from source4/torture.  

> If we're going to remove the distinction in the names, then we should also
> reconcile the underlying code.
> 
> > The different files used to matter when we had two different build
> > systems, because we ran one or the other, and then later ran either both
> > or only the source3 files. 
> I think they still somewhat matter for the reasons mentioned above.
> 
> > However, in each of these files, tests are run against arbitrary
> > environments and both call smbtorture for much of the work.  That means
> > the name isn't really a reflection of if the test is a 'samba3' or
> > 'samba4' test any more.  The primary difference that I'm aware of is
> > that the --target={samba3,samba4} can change some behaviours in the
> > test.
> The --target parameter is supposed to match the environment (e.g. "samba3" for
> "s3dc"), not the origin of the test itself (which is what the samba3/samba4 prefix
> represents).

Except that isn't the case - the code that puts the samba3 in the
command line of the smbtorture binary is in source3/selftest/tests.py.
Both AD DC and file-server environments are tested with that set.

> > We should also try and move our multitude of tests.py files to either
> > one file, or split it out in some reasonable way, as currently the
> > allocation is essentially historic. 
> I think once we've merged a specific bit of code, that test can be moved to
> the top-level tests.py.
> 
> > As a start, perhaps all the python tests should move up to
> > selftest/tests.py?
> That's a good idea.
> 
> Cheers,
> 
> Jelmer
> 

-- 
Andrew Bartlett
http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba