Test infrastructure improvements

[Jelmer Vernooij]

On Sat, Oct 25, 2014 at 03:06:48PM -0700, Jelmer Vernooij wrote:
> Matthieu and I got together a couple of weeks ago to hack on the
> Samba testsuite.
> 
> Our main goal has so far been on reducing the overall run time of the
> testsuite, through various means. Matthieu has been pesking me about
> running tests in parallel for a long time.
> 
> I've put the todo-list we compiled on the wiki:
> https://wiki.samba.org/index.php/Test_Infrastructure_Sprint
> 
> What we have at the moment:
> 
> * drop the samba3/samba4 distinction?
I think this is a much larger change, since it ties in with other code changes
to e.g. remove the distinction between smbtorture3 and smbtorture4.

Perhaps we should hold off on this as part of the test infrastructure
improvements, though I agree it would be a nice thing to do in general.

> * randomize test order [mat]
Any news on this?

> * statistics!
>  * measure total run time and per test run time
>  * prevent slow test addition
>  * record env startup time [jelmer]
> * make testsuite output top X [mat]
This is done :)

It would be good to improve the formatting further. At the moment, if I just run one test I get 9 empty lines.
Can we output the unit of the test durations? (e.g. seconds)

> * reduce the cardinality of RPC tests
Nothing has happened on this as far as I know.

> * parallelize with testr [jelmer]
>  * migrate to subunit2
This is in progress; I've got some changes that update our in-tree subunit to
version 2. We'll still use subunit1 output internally (by converting v2 output
to v1 before we parsing it).

>  * make all tests support --list
Still some work to do here, especially for the pidl tap tests.

>  * reduce selftest run overhead
> * documentation
> * simplification of test run infrastructure

Cheers,

Jelmer