Test infrastructure improvements

[Jelmer Vernooij]

On Mon, Oct 27, 2014 at 06:35:29PM +1300, Andrew Bartlett wrote:
> On Sun, 2014-10-26 at 12:07 +0100, Michael Adam wrote:
> > On 2014-10-26 at 07:21 +0100, Jelmer Vernooij wrote:
> > > On Sat, Oct 25, 2014 at 09:26:44PM -0700, Matthieu Patou wrote:
> > > > On 10/25/2014 06:01 PM, Michael Adam wrote:
> > > > >On 2014-10-25 at 15:06 -0700, Jelmer Vernooij wrote:
> > > > >If I look into st/subunit, I see this:
> > > > >
> > > > >~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > > > >progress: 1
> > > > >time: 2014-10-26 02:18:18.000000Z
> > > > >testsuite: samba.tests.param
> > > > >progress: push
> > > > >time: 2014-10-26 02:18:18.000000Z
> > > > ><B3>)^A5,samba.tests.param.LoadParmTestCase.test_init<C3>(<86>C<B3>)^A<3samba.tests.param.LoadParmTestCase.test_is_mydomain^\,.<F4><B3>)^A:1samba.tests.param.LoadParmTestCase.test_is_myname<95>6$<91><B3>)^A7.samba.tests.param.LoadParmTestCase.test_length<9B>W<F4>ܳ)^A=4samba.tests.param.LoadParmTestCase.test_load_default<B5><F7><9A>O<B3>)^A at E;samba.tests.param.LoadParmTestCase.test_section_nonexistent<A8><B7>^Y<99><B3>)^A>5samba.tests.param.LoadParmTestCase.test_set_workgroupF<EC>^L^V<B3>+^B=TL=<CB>
> > > > ><U+0081><AD><88>,samba.tests.param.LoadParmTestCase.test_init<F6><EA><A9>`<B3>+^C=TL=<CB>£^<88>,samba.tests.param.LoadParmTestCase.test_init<FA>\Ma<B3>+^B at ETL=<CB>¥<9C><C0>3samba.tests.param.LoadParmTestCase.test_is_mydomain"<CD>5M<B3>+^C at ETL=<CB>¦<E0>
> > > > ><F8>3samba.tests.param.LoadParmTestCase.test_is_mydomainC<C3>^N^^<B3>+^B at CTL=<CB>¨^Q<A8>1samba.tests.param.LoadParmTestCase.test_is_myname^A<g<B1><B3>+^C at CTL=<CB>©^_01samba.tests.param.LoadParmTestCase.test_is_mynama<F5><B4><B3>+^B?TL=<CB>ªH^P.samba.tests.param.LoadParmTestCase.test_length<99><A4><EA>ٳ+^C?TL=<CB>«U<98>.samba.tests.param.LoadParmTestCase.test_length^L6o<80><B3>+
> > > > >^B at FTL=<CB>¬n<D8>4samba.tests.param.LoadParmTestCase.test_load_default<9E>:<B3>W<B3>+^C at FTL=<CB>±T<C0>4samba.tests.param.LoadParmTestCase.test_load_defaultڢ^E<B4><B3>+^B at MTL=<CB>²y<B8>;samba.tests.param.LoadParmTestCase.test_section_nonexistent^]x^Z<E6>
> > > > ><B3>+^C at MTL=<CB>¸m(;samba.tests.param.LoadParmTestCase.test_section_nonexistent<B8>snʳ+^B at GTL=<CB>¹<AD>x5samba.tests.param.LoadParmTestCase.test_set_workgroup<EC>}<FF>^X<B3>+^C at GTL=<CB>º<B3>05samba.tests.param.LoadParmTestCase.test_set_workgroup<C0>#a
> > > > ><A5>time: 2014-10-26 02:18:19.000000
> > > > >progress: pop
> > > > >command: python -m subunit.run $LISTOPT samba.tests.param
> > > > >expanded command: python -m subunit.run $LISTOPT
> > > > >samba.tests.param
> > > > >testsuite-success: samba.tests.param
> > > > >~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > > > I can try to figure it out.
> > > This is happening because we try to use the system subunit if possible. However, you have a
> > > very new subunit installed, which produces subunit2. The Samba subunit parser (which is
> > > custom and in perl) only supports subunitv1 and barfs on this.
> > 
> > Ok, but what does that mean?
> > 
> > If I get it right, the test result as such is not affected,
> > only the display of the results, right?
> > 
> > And is it really so? Volker told me about problems he
> > had with the python/samba/tests/messaging test that
> > reliably failed for him with supposedly innocuous changes
> > top messging, when run from make test, but succeeded when
> > run manually. (If I recall everything correctly.)
> > 
> > And are there plans to fix it?
> > 
> > If I can help anywhere, let me know.
> 
> I've had similar issues, at the time I was debugging and developing the
> bad password lockout tests.  Removing the subunit2 packages from the
> system was the fix. 
> 
> Also remember that outside make test you need to run the via a subunit
> filter to get sane output.  (This will apply even more after the full
> move to subunit2).

I've sent you a patchset that should make us cope with newer versions of
subunit version 2.

My plan is to provide some more documentation around subunit and ways to use it
in Samba once we get our current set of test infrastructure improvements done.

Cheers,

Jelmer