Test infrastructure improvements

[Matthieu Patou]

On 10/25/2014 06:01 PM, Michael Adam wrote:
> On 2014-10-25 at 15:06 -0700, Jelmer Vernooij wrote:
>> Matthieu and I got together a couple of weeks ago to hack on the
>> Samba testsuite.
>>
>> Our main goal has so far been on reducing the overall run time of the
>> testsuite, through various means. Matthieu has been pesking me about
>> running tests in parallel for a long time.
>>
>> I've put the todo-list we compiled on the wiki:
>> https://wiki.samba.org/index.php/Test_Infrastructure_Sprint
>>
>> What we have at the moment:
>>
>> * drop the samba3/samba4 distinction?
>> * randomize test order [mat]
>> * statistics!
>>   * measure total run time and per test run time
>>   * prevent slow test addition
>>   * record env startup time [jelmer]
>> * make testsuite output top X [mat]
>> * reduce the cardinality of RPC tests
>> * parallelize with testr [jelmer]
>>   * migrate to subunit2
>>   * make all tests support --list
>>   * <s>require all tests output subunit</s>
>>   * reduce selftest run overhead
>>    * <s>make symbol deduplication a test, not a build phase</s>
>> * documentation
>> * simplification of test run infrastructure
> Nice progress!
>
>
> I noticed a problem with the tests in python/samba/tests:
> They don't seem to produce parseable output (instead of
> newlines there is garbage). I noticed due to the top 10
> timer output showing negative numbers:
>
> e.g.
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> $make test TESTS=samba.tests.param
> ...
> [1/1 at 0s] samba.tests.param
>
> ALL OK (0 tests in 1 testsuites)
>
> A summary with detailed information can be found in:
>    ./st/summary
> TOP 10 slowest tests
> samba.tests.param -> -1414289898
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> If I look into st/subunit, I see this:
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> progress: 1
> time: 2014-10-26 02:18:18.000000Z
> testsuite: samba.tests.param
> progress: push
> time: 2014-10-26 02:18:18.000000Z
> <B3>)^A5,samba.tests.param.LoadParmTestCase.test_init<C3>(<86>C<B3>)^A<3samba.tests.param.LoadParmTestCase.test_is_mydomain^\,.<F4><B3>)^A:1samba.tests.param.LoadParmTestCase.test_is_myname<95>6$<91><B3>)^A7.samba.tests.param.LoadParmTestCase.test_length<9B>W<F4>ܳ)^A=4samba.tests.param.LoadParmTestCase.test_load_default<B5><F7><9A>O<B3>)^A at E;samba.tests.param.LoadParmTestCase.test_section_nonexistent<A8><B7>^Y<99><B3>)^A>5samba.tests.param.LoadParmTestCase.test_set_workgroupF<EC>^L^V<B3>+^B=TL=<CB>
> <U+0081><AD><88>,samba.tests.param.LoadParmTestCase.test_init<F6><EA><A9>`<B3>+^C=TL=<CB>£^<88>,samba.tests.param.LoadParmTestCase.test_init<FA>\Ma<B3>+^B at ETL=<CB>¥<9C><C0>3samba.tests.param.LoadParmTestCase.test_is_mydomain"<CD>5M<B3>+^C at ETL=<CB>¦<E0>
> <F8>3samba.tests.param.LoadParmTestCase.test_is_mydomainC<C3>^N^^<B3>+^B at CTL=<CB>¨^Q<A8>1samba.tests.param.LoadParmTestCase.test_is_myname^A<g<B1><B3>+^C at CTL=<CB>©^_01samba.tests.param.LoadParmTestCase.test_is_mynama<F5><B4><B3>+^B?TL=<CB>ªH^P.samba.tests.param.LoadParmTestCase.test_length<99><A4><EA>ٳ+^C?TL=<CB>«U<98>.samba.tests.param.LoadParmTestCase.test_length^L6o<80><B3>+
> ^B at FTL=<CB>¬n<D8>4samba.tests.param.LoadParmTestCase.test_load_default<9E>:<B3>W<B3>+^C at FTL=<CB>±T<C0>4samba.tests.param.LoadParmTestCase.test_load_defaultڢ^E<B4><B3>+^B at MTL=<CB>²y<B8>;samba.tests.param.LoadParmTestCase.test_section_nonexistent^]x^Z<E6>
> <B3>+^C at MTL=<CB>¸m(;samba.tests.param.LoadParmTestCase.test_section_nonexistent<B8>snʳ+^B at GTL=<CB>¹<AD>x5samba.tests.param.LoadParmTestCase.test_set_workgroup<EC>}<FF>^X<B3>+^C at GTL=<CB>º<B3>05samba.tests.param.LoadParmTestCase.test_set_workgroup<C0>#a
> <A5>time: 2014-10-26 02:18:19.000000
> progress: pop
> command: python -m subunit.run $LISTOPT samba.tests.param
> expanded command: python -m subunit.run $LISTOPT
> samba.tests.param
> testsuite-success: samba.tests.param
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I can try to figure it out.
Matthieu.