Test infrastructure improvements

[Jelmer Vernooij]

On Sat, Oct 25, 2014 at 09:26:44PM -0700, Matthieu Patou wrote:
> On 10/25/2014 06:01 PM, Michael Adam wrote:
> >On 2014-10-25 at 15:06 -0700, Jelmer Vernooij wrote:
> >>Matthieu and I got together a couple of weeks ago to hack on the
> >>Samba testsuite.
> >>
> >>Our main goal has so far been on reducing the overall run time of the
> >>testsuite, through various means. Matthieu has been pesking me about
> >>running tests in parallel for a long time.
> >>
> >>I've put the todo-list we compiled on the wiki:
> >>https://wiki.samba.org/index.php/Test_Infrastructure_Sprint
> >>
> >>What we have at the moment:
> >>
> >>* drop the samba3/samba4 distinction?
> >>* randomize test order [mat]
> >>* statistics!
> >>  * measure total run time and per test run time
> >>  * prevent slow test addition
> >>  * record env startup time [jelmer]
> >>* make testsuite output top X [mat]
> >>* reduce the cardinality of RPC tests
> >>* parallelize with testr [jelmer]
> >>  * migrate to subunit2
> >>  * make all tests support --list
> >>  * <s>require all tests output subunit</s>
> >>  * reduce selftest run overhead
> >>   * <s>make symbol deduplication a test, not a build phase</s>
> >>* documentation
> >>* simplification of test run infrastructure
> >Nice progress!
> >
> >
> >I noticed a problem with the tests in python/samba/tests:
> >They don't seem to produce parseable output (instead of
> >newlines there is garbage). I noticed due to the top 10
> >timer output showing negative numbers:
> >
> >e.g.
> >~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >$make test TESTS=samba.tests.param
> >...
> >[1/1 at 0s] samba.tests.param
> >
> >ALL OK (0 tests in 1 testsuites)
> >
> >A summary with detailed information can be found in:
> >   ./st/summary
> >TOP 10 slowest tests
> >samba.tests.param -> -1414289898
> >~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> >If I look into st/subunit, I see this:
> >
> >~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >progress: 1
> >time: 2014-10-26 02:18:18.000000Z
> >testsuite: samba.tests.param
> >progress: push
> >time: 2014-10-26 02:18:18.000000Z
> ><B3>)^A5,samba.tests.param.LoadParmTestCase.test_init<C3>(<86>C<B3>)^A<3samba.tests.param.LoadParmTestCase.test_is_mydomain^\,.<F4><B3>)^A:1samba.tests.param.LoadParmTestCase.test_is_myname<95>6$<91><B3>)^A7.samba.tests.param.LoadParmTestCase.test_length<9B>W<F4>ܳ)^A=4samba.tests.param.LoadParmTestCase.test_load_default<B5><F7><9A>O<B3>)^A at E;samba.tests.param.LoadParmTestCase.test_section_nonexistent<A8><B7>^Y<99><B3>)^A>5samba.tests.param.LoadParmTestCase.test_set_workgroupF<EC>^L^V<B3>+^B=TL=<CB>
> ><U+0081><AD><88>,samba.tests.param.LoadParmTestCase.test_init<F6><EA><A9>`<B3>+^C=TL=<CB>£^<88>,samba.tests.param.LoadParmTestCase.test_init<FA>\Ma<B3>+^B at ETL=<CB>¥<9C><C0>3samba.tests.param.LoadParmTestCase.test_is_mydomain"<CD>5M<B3>+^C at ETL=<CB>¦<E0>
> ><F8>3samba.tests.param.LoadParmTestCase.test_is_mydomainC<C3>^N^^<B3>+^B at CTL=<CB>¨^Q<A8>1samba.tests.param.LoadParmTestCase.test_is_myname^A<g<B1><B3>+^C at CTL=<CB>©^_01samba.tests.param.LoadParmTestCase.test_is_mynama<F5><B4><B3>+^B?TL=<CB>ªH^P.samba.tests.param.LoadParmTestCase.test_length<99><A4><EA>ٳ+^C?TL=<CB>«U<98>.samba.tests.param.LoadParmTestCase.test_length^L6o<80><B3>+
> >^B at FTL=<CB>¬n<D8>4samba.tests.param.LoadParmTestCase.test_load_default<9E>:<B3>W<B3>+^C at FTL=<CB>±T<C0>4samba.tests.param.LoadParmTestCase.test_load_defaultڢ^E<B4><B3>+^B at MTL=<CB>²y<B8>;samba.tests.param.LoadParmTestCase.test_section_nonexistent^]x^Z<E6>
> ><B3>+^C at MTL=<CB>¸m(;samba.tests.param.LoadParmTestCase.test_section_nonexistent<B8>snʳ+^B at GTL=<CB>¹<AD>x5samba.tests.param.LoadParmTestCase.test_set_workgroup<EC>}<FF>^X<B3>+^C at GTL=<CB>º<B3>05samba.tests.param.LoadParmTestCase.test_set_workgroup<C0>#a
> ><A5>time: 2014-10-26 02:18:19.000000
> >progress: pop
> >command: python -m subunit.run $LISTOPT samba.tests.param
> >expanded command: python -m subunit.run $LISTOPT
> >samba.tests.param
> >testsuite-success: samba.tests.param
> >~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> I can try to figure it out.
This is happening because we try to use the system subunit if possible. However, you have a
very new subunit installed, which produces subunit2. The Samba subunit parser (which is
custom and in perl) only supports subunitv1 and barfs on this.

Cheers,

Jelmer