Test infrastructure improvements

[Andrew Bartlett]

On Sun, 2014-10-26 at 12:07 +0100, Michael Adam wrote:
> On 2014-10-26 at 07:21 +0100, Jelmer Vernooij wrote:
> > On Sat, Oct 25, 2014 at 09:26:44PM -0700, Matthieu Patou wrote:
> > > On 10/25/2014 06:01 PM, Michael Adam wrote:
> > > >On 2014-10-25 at 15:06 -0700, Jelmer Vernooij wrote:
> > > >If I look into st/subunit, I see this:
> > > >
> > > >~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > > >progress: 1
> > > >time: 2014-10-26 02:18:18.000000Z
> > > >testsuite: samba.tests.param
> > > >progress: push
> > > >time: 2014-10-26 02:18:18.000000Z
> > > ><B3>)^A5,samba.tests.param.LoadParmTestCase.test_init<C3>(<86>C<B3>)^A<3samba.tests.param.LoadParmTestCase.test_is_mydomain^\,.<F4><B3>)^A:1samba.tests.param.LoadParmTestCase.test_is_myname<95>6$<91><B3>)^A7.samba.tests.param.LoadParmTestCase.test_length<9B>W<F4>ܳ)^A=4samba.tests.param.LoadParmTestCase.test_load_default<B5><F7><9A>O<B3>)^A at E;samba.tests.param.LoadParmTestCase.test_section_nonexistent<A8><B7>^Y<99><B3>)^A>5samba.tests.param.LoadParmTestCase.test_set_workgroupF<EC>^L^V<B3>+^B=TL=<CB>
> > > ><U+0081><AD><88>,samba.tests.param.LoadParmTestCase.test_init<F6><EA><A9>`<B3>+^C=TL=<CB>£^<88>,samba.tests.param.LoadParmTestCase.test_init<FA>\Ma<B3>+^B at ETL=<CB>¥<9C><C0>3samba.tests.param.LoadParmTestCase.test_is_mydomain"<CD>5M<B3>+^C at ETL=<CB>¦<E0>
> > > ><F8>3samba.tests.param.LoadParmTestCase.test_is_mydomainC<C3>^N^^<B3>+^B at CTL=<CB>¨^Q<A8>1samba.tests.param.LoadParmTestCase.test_is_myname^A<g<B1><B3>+^C at CTL=<CB>©^_01samba.tests.param.LoadParmTestCase.test_is_mynama<F5><B4><B3>+^B?TL=<CB>ªH^P.samba.tests.param.LoadParmTestCase.test_length<99><A4><EA>ٳ+^C?TL=<CB>«U<98>.samba.tests.param.LoadParmTestCase.test_length^L6o<80><B3>+
> > > >^B at FTL=<CB>¬n<D8>4samba.tests.param.LoadParmTestCase.test_load_default<9E>:<B3>W<B3>+^C at FTL=<CB>±T<C0>4samba.tests.param.LoadParmTestCase.test_load_defaultڢ^E<B4><B3>+^B at MTL=<CB>²y<B8>;samba.tests.param.LoadParmTestCase.test_section_nonexistent^]x^Z<E6>
> > > ><B3>+^C at MTL=<CB>¸m(;samba.tests.param.LoadParmTestCase.test_section_nonexistent<B8>snʳ+^B at GTL=<CB>¹<AD>x5samba.tests.param.LoadParmTestCase.test_set_workgroup<EC>}<FF>^X<B3>+^C at GTL=<CB>º<B3>05samba.tests.param.LoadParmTestCase.test_set_workgroup<C0>#a
> > > ><A5>time: 2014-10-26 02:18:19.000000
> > > >progress: pop
> > > >command: python -m subunit.run $LISTOPT samba.tests.param
> > > >expanded command: python -m subunit.run $LISTOPT
> > > >samba.tests.param
> > > >testsuite-success: samba.tests.param
> > > >~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > > I can try to figure it out.
> > This is happening because we try to use the system subunit if possible. However, you have a
> > very new subunit installed, which produces subunit2. The Samba subunit parser (which is
> > custom and in perl) only supports subunitv1 and barfs on this.
> 
> Ok, but what does that mean?
> 
> If I get it right, the test result as such is not affected,
> only the display of the results, right?
> 
> And is it really so? Volker told me about problems he
> had with the python/samba/tests/messaging test that
> reliably failed for him with supposedly innocuous changes
> top messging, when run from make test, but succeeded when
> run manually. (If I recall everything correctly.)
> 
> And are there plans to fix it?
> 
> If I can help anywhere, let me know.

I've had similar issues, at the time I was debugging and developing the
bad password lockout tests.  Removing the subunit2 packages from the
system was the fix. 

Also remember that outside make test you need to run the via a subunit
filter to get sane output.  (This will apply even more after the full
move to subunit2).

Andrew Bartlett

-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba