Test infrastructure improvements

[Michael Adam]

On 2014-10-25 at 15:06 -0700, Jelmer Vernooij wrote:
> Matthieu and I got together a couple of weeks ago to hack on the
> Samba testsuite.
> 
> Our main goal has so far been on reducing the overall run time of the
> testsuite, through various means. Matthieu has been pesking me about
> running tests in parallel for a long time.
> 
> I've put the todo-list we compiled on the wiki:
> https://wiki.samba.org/index.php/Test_Infrastructure_Sprint
> 
> What we have at the moment:
> 
> * drop the samba3/samba4 distinction?
> * randomize test order [mat]
> * statistics!
>  * measure total run time and per test run time
>  * prevent slow test addition
>  * record env startup time [jelmer]
> * make testsuite output top X [mat]
> * reduce the cardinality of RPC tests
> * parallelize with testr [jelmer]
>  * migrate to subunit2
>  * make all tests support --list
>  * <s>require all tests output subunit</s>
>  * reduce selftest run overhead
>   * <s>make symbol deduplication a test, not a build phase</s>
> * documentation
> * simplification of test run infrastructure

Nice progress!


I noticed a problem with the tests in python/samba/tests:
They don't seem to produce parseable output (instead of
newlines there is garbage). I noticed due to the top 10
timer output showing negative numbers:

e.g.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
$make test TESTS=samba.tests.param
...
[1/1 at 0s] samba.tests.param

ALL OK (0 tests in 1 testsuites)

A summary with detailed information can be found in:
  ./st/summary
TOP 10 slowest tests
samba.tests.param -> -1414289898
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If I look into st/subunit, I see this:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
progress: 1
time: 2014-10-26 02:18:18.000000Z
testsuite: samba.tests.param
progress: push
time: 2014-10-26 02:18:18.000000Z
<B3>)^A5,samba.tests.param.LoadParmTestCase.test_init<C3>(<86>C<B3>)^A<3samba.tests.param.LoadParmTestCase.test_is_mydomain^\,.<F4><B3>)^A:1samba.tests.param.LoadParmTestCase.test_is_myname<95>6$<91><B3>)^A7.samba.tests.param.LoadParmTestCase.test_length<9B>W<F4>ܳ)^A=4samba.tests.param.LoadParmTestCase.test_load_default<B5><F7><9A>O<B3>)^A at E;samba.tests.param.LoadParmTestCase.test_section_nonexistent<A8><B7>^Y<99><B3>)^A>5samba.tests.param.LoadParmTestCase.test_set_workgroupF<EC>^L^V<B3>+^B=TL=<CB>
<U+0081><AD><88>,samba.tests.param.LoadParmTestCase.test_init<F6><EA><A9>`<B3>+^C=TL=<CB>£^<88>,samba.tests.param.LoadParmTestCase.test_init<FA>\Ma<B3>+^B at ETL=<CB>¥<9C><C0>3samba.tests.param.LoadParmTestCase.test_is_mydomain"<CD>5M<B3>+^C at ETL=<CB>¦<E0>
<F8>3samba.tests.param.LoadParmTestCase.test_is_mydomainC<C3>^N^^<B3>+^B at CTL=<CB>¨^Q<A8>1samba.tests.param.LoadParmTestCase.test_is_myname^A<g<B1><B3>+^C at CTL=<CB>©^_01samba.tests.param.LoadParmTestCase.test_is_mynama<F5><B4><B3>+^B?TL=<CB>ªH^P.samba.tests.param.LoadParmTestCase.test_length<99><A4><EA>ٳ+^C?TL=<CB>«U<98>.samba.tests.param.LoadParmTestCase.test_length^L6o<80><B3>+
^B at FTL=<CB>¬n<D8>4samba.tests.param.LoadParmTestCase.test_load_default<9E>:<B3>W<B3>+^C at FTL=<CB>±T<C0>4samba.tests.param.LoadParmTestCase.test_load_defaultڢ^E<B4><B3>+^B at MTL=<CB>²y<B8>;samba.tests.param.LoadParmTestCase.test_section_nonexistent^]x^Z<E6>
<B3>+^C at MTL=<CB>¸m(;samba.tests.param.LoadParmTestCase.test_section_nonexistent<B8>snʳ+^B at GTL=<CB>¹<AD>x5samba.tests.param.LoadParmTestCase.test_set_workgroup<EC>}<FF>^X<B3>+^C at GTL=<CB>º<B3>05samba.tests.param.LoadParmTestCase.test_set_workgroup<C0>#a
<A5>time: 2014-10-26 02:18:19.000000
progress: pop
command: python -m subunit.run $LISTOPT samba.tests.param
expanded command: python -m subunit.run $LISTOPT
samba.tests.param
testsuite-success: samba.tests.param
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Therefore the end-time is not caught.
It seems to list the test cases run, but with random garbage
inbetween.

This is the same with all the tests in the python/samba/tests
directory.

I tried to understand what happens here, but with all these levels
of python tests, pyfoo.c, testtools and such, it is not really
easy to understand (for me) where this output is generated.
So a give up for now and report it in the hope that someone else
can easily tell what's going on. :-)

Cheers - Michael


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20141026/d458dfe6/attachment.pgp>