Test infrastructure improvements

[Andrew Bartlett]

On Sat, 2014-11-01 at 17:15 +0100, Jelmer Vernooij wrote:
> On Mon, Oct 27, 2014 at 06:35:29PM +1300, Andrew Bartlett wrote:
> > On Sun, 2014-10-26 at 12:07 +0100, Michael Adam wrote:
> > > On 2014-10-26 at 07:21 +0100, Jelmer Vernooij wrote:
> > > > On Sat, Oct 25, 2014 at 09:26:44PM -0700, Matthieu Patou wrote:
> > > > > On 10/25/2014 06:01 PM, Michael Adam wrote:
> > > > > >On 2014-10-25 at 15:06 -0700, Jelmer Vernooij wrote:
> > > > > >If I look into st/subunit, I see this:
> > > > > >
> > > > > >~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > > > > >progress: 1
> > > > > >time: 2014-10-26 02:18:18.000000Z
> > > > > >testsuite: samba.tests.param
> > > > > >progress: push
> > > > > >time: 2014-10-26 02:18:18.000000Z
> > > > > ><B3>)^A5,samba.tests.param.LoadParmTestCase.test_init<C3>(<86>C<B3>)^A<3samba.tests.param.LoadParmTestCase.test_is_mydomain^\,.<F4><B3>)^A:1samba.tests.param.LoadParmTestCase.test_is_myname<95>6$<91><B3>)^A7.samba.tests.param.LoadParmTestCase.test_length<9B>W<F4>ܳ)^A=4samba.tests.param.LoadParmTestCase.test_load_default<B5><F7><9A>O<B3>)^A at E;samba.tests.param.LoadParmTestCase.test_section_nonexistent<A8><B7>^Y<99><B3>)^A>5samba.tests.param.LoadParmTestCase.test_set_workgroupF<EC>^L^V<B3>+^B=TL=<CB>
> > > > > ><U+0081><AD><88>,samba.tests.param.LoadParmTestCase.test_init<F6><EA><A9>`<B3>+^C=TL=<CB>£^<88>,samba.tests.param.LoadParmTestCase.test_init<FA>\Ma<B3>+^B at ETL=<CB>¥<9C><C0>3samba.tests.param.LoadParmTestCase.test_is_mydomain"<CD>5M<B3>+^C at ETL=<CB>¦<E0>
> > > > > ><F8>3samba.tests.param.LoadParmTestCase.test_is_mydomainC<C3>^N^^<B3>+^B at CTL=<CB>¨^Q<A8>1samba.tests.param.LoadParmTestCase.test_is_myname^A<g<B1><B3>+^C at CTL=<CB>©^_01samba.tests.param.LoadParmTestCase.test_is_mynama<F5><B4><B3>+^B?TL=<CB>ªH^P.samba.tests.param.LoadParmTestCase.test_length<99><A4><EA>ٳ+^C?TL=<CB>«U<98>.samba.tests.param.LoadParmTestCase.test_length^L6o<80><B3>+
> > > > > >^B at FTL=<CB>¬n<D8>4samba.tests.param.LoadParmTestCase.test_load_default<9E>:<B3>W<B3>+^C at FTL=<CB>±T<C0>4samba.tests.param.LoadParmTestCase.test_load_defaultڢ^E<B4><B3>+^B at MTL=<CB>²y<B8>;samba.tests.param.LoadParmTestCase.test_section_nonexistent^]x^Z<E6>
> > > > > ><B3>+^C at MTL=<CB>¸m(;samba.tests.param.LoadParmTestCase.test_section_nonexistent<B8>snʳ+^B at GTL=<CB>¹<AD>x5samba.tests.param.LoadParmTestCase.test_set_workgroup<EC>}<FF>^X<B3>+^C at GTL=<CB>º<B3>05samba.tests.param.LoadParmTestCase.test_set_workgroup<C0>#a
> > > > > ><A5>time: 2014-10-26 02:18:19.000000
> > > > > >progress: pop
> > > > > >command: python -m subunit.run $LISTOPT samba.tests.param
> > > > > >expanded command: python -m subunit.run $LISTOPT
> > > > > >samba.tests.param
> > > > > >testsuite-success: samba.tests.param
> > > > > >~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > > > > I can try to figure it out.
> > > > This is happening because we try to use the system subunit if possible. However, you have a
> > > > very new subunit installed, which produces subunit2. The Samba subunit parser (which is
> > > > custom and in perl) only supports subunitv1 and barfs on this.
> > > 
> > > Ok, but what does that mean?
> > > 
> > > If I get it right, the test result as such is not affected,
> > > only the display of the results, right?
> > > 
> > > And is it really so? Volker told me about problems he
> > > had with the python/samba/tests/messaging test that
> > > reliably failed for him with supposedly innocuous changes
> > > top messging, when run from make test, but succeeded when
> > > run manually. (If I recall everything correctly.)
> > > 
> > > And are there plans to fix it?
> > > 
> > > If I can help anywhere, let me know.
> > 
> > I've had similar issues, at the time I was debugging and developing the
> > bad password lockout tests.  Removing the subunit2 packages from the
> > system was the fix. 
> > 
> > Also remember that outside make test you need to run the via a subunit
> > filter to get sane output.  (This will apply even more after the full
> > move to subunit2).
> 
> I've sent you a patchset that should make us cope with newer versions of
> subunit version 2.

Can I have that as a git tree on samba.org?  I'll be comparing it with
the tree I build last time to see the delta, but it gives me a bit more
confidence with such a large set of patches when I can fetch it over SSH
rather than unsigned mails.  Also, can we improve the import script to
keep a GIT hash of the import, for later verification?  I re-did the
import and checked the delta, but it just fetched HEAD of all those
trees as far as I can tell.  (I also actually read the diff.)

Thanks,

Andrew Bartlett

-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba