Test infrastructure improvements

[Jelmer Vernooij]

On Sun, Nov 02, 2014 at 12:27:11PM +1300, Andrew Bartlett wrote:
> On Sat, 2014-11-01 at 17:15 +0100, Jelmer Vernooij wrote:
> > On Mon, Oct 27, 2014 at 06:35:29PM +1300, Andrew Bartlett wrote:
> > > On Sun, 2014-10-26 at 12:07 +0100, Michael Adam wrote:
> > > > On 2014-10-26 at 07:21 +0100, Jelmer Vernooij wrote:
> > > > > On Sat, Oct 25, 2014 at 09:26:44PM -0700, Matthieu Patou wrote:
> > > > > > On 10/25/2014 06:01 PM, Michael Adam wrote:
> > > > > > >On 2014-10-25 at 15:06 -0700, Jelmer Vernooij wrote:
> > > > > > >If I look into st/subunit, I see this:
> > > > > > >
> > > > > > >~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > > > > > >progress: 1
> > > > > > >time: 2014-10-26 02:18:18.000000Z
> > > > > > >testsuite: samba.tests.param
> > > > > > >progress: push
> > > > > > >time: 2014-10-26 02:18:18.000000Z
> > > > > > ><B3>)^A5,samba.tests.param.LoadParmTestCase.test_init<C3>(<86>C<B3>)^A<3samba.tests.param.LoadParmTestCase.test_is_mydomain^\,.<F4><B3>)^A:1samba.tests.param.LoadParmTestCase.test_is_myname<95>6$<91><B3>)^A7.samba.tests.param.LoadParmTestCase.test_length<9B>W<F4>ܳ)^A=4samba.tests.param.LoadParmTestCase.test_load_default<B5><F7><9A>O<B3>)^A at E;samba.tests.param.LoadParmTestCase.test_section_nonexistent<A8><B7>^Y<99><B3>)^A>5samba.tests.param.LoadParmTestCase.test_set_workgroupF<EC>^L^V<B3>+^B=TL=<CB>
> > > > > > ><U+0081><AD><88>,samba.tests.param.LoadParmTestCase.test_init<F6><EA><A9>`<B3>+^C=TL=<CB>£^<88>,samba.tests.param.LoadParmTestCase.test_init<FA>\Ma<B3>+^B at ETL=<CB>¥<9C><C0>3samba.tests.param.LoadParmTestCase.test_is_mydomain"<CD>5M<B3>+^C at ETL=<CB>¦<E0>
> > > > > > ><F8>3samba.tests.param.LoadParmTestCase.test_is_mydomainC<C3>^N^^<B3>+^B at CTL=<CB>¨^Q<A8>1samba.tests.param.LoadParmTestCase.test_is_myname^A<g<B1><B3>+^C at CTL=<CB>©^_01samba.tests.param.LoadParmTestCase.test_is_mynama<F5><B4><B3>+^B?TL=<CB>ªH^P.samba.tests.param.LoadParmTestCase.test_length<99><A4><EA>ٳ+^C?TL=<CB>«U<98>.samba.tests.param.LoadParmTestCase.test_length^L6o<80><B3>+
> > > > > > >^B at FTL=<CB>¬n<D8>4samba.tests.param.LoadParmTestCase.test_load_default<9E>:<B3>W<B3>+^C at FTL=<CB>±T<C0>4samba.tests.param.LoadParmTestCase.test_load_defaultڢ^E<B4><B3>+^B at MTL=<CB>²y<B8>;samba.tests.param.LoadParmTestCase.test_section_nonexistent^]x^Z<E6>
> > > > > > ><B3>+^C at MTL=<CB>¸m(;samba.tests.param.LoadParmTestCase.test_section_nonexistent<B8>snʳ+^B at GTL=<CB>¹<AD>x5samba.tests.param.LoadParmTestCase.test_set_workgroup<EC>}<FF>^X<B3>+^C at GTL=<CB>º<B3>05samba.tests.param.LoadParmTestCase.test_set_workgroup<C0>#a
> > > > > > ><A5>time: 2014-10-26 02:18:19.000000
> > > > > > >progress: pop
> > > > > > >command: python -m subunit.run $LISTOPT samba.tests.param
> > > > > > >expanded command: python -m subunit.run $LISTOPT
> > > > > > >samba.tests.param
> > > > > > >testsuite-success: samba.tests.param
> > > > > > >~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > > > > > I can try to figure it out.
> > > > > This is happening because we try to use the system subunit if possible. However, you have a
> > > > > very new subunit installed, which produces subunit2. The Samba subunit parser (which is
> > > > > custom and in perl) only supports subunitv1 and barfs on this.
> > > > 
> > > > Ok, but what does that mean?
> > > > 
> > > > If I get it right, the test result as such is not affected,
> > > > only the display of the results, right?
> > > > 
> > > > And is it really so? Volker told me about problems he
> > > > had with the python/samba/tests/messaging test that
> > > > reliably failed for him with supposedly innocuous changes
> > > > top messging, when run from make test, but succeeded when
> > > > run manually. (If I recall everything correctly.)
> > > > 
> > > > And are there plans to fix it?
> > > > 
> > > > If I can help anywhere, let me know.
> > > 
> > > I've had similar issues, at the time I was debugging and developing the
> > > bad password lockout tests.  Removing the subunit2 packages from the
> > > system was the fix. 
> > > 
> > > Also remember that outside make test you need to run the via a subunit
> > > filter to get sane output.  (This will apply even more after the full
> > > move to subunit2).
> > 
> > I've sent you a patchset that should make us cope with newer versions of
> > subunit version 2.
> 
> Can I have that as a git tree on samba.org?  I'll be comparing it with
> the tree I build last time to see the delta, but it gives me a bit more
> confidence with such a large set of patches when I can fetch it over SSH
> rather than unsigned mails.  Also, can we improve the import script to
> keep a GIT hash of the import, for later verification?  I re-did the
> import and checked the delta, but it just fetched HEAD of all those
> trees as far as I can tell.  (I also actually read the diff.)

git.samba.org appears to be down at the moment, ssh-wise, so I've pushed my branch
to 'subunit2' on git://github.com/jelmer/samba

I'd like to use submodules for the third party projects. That would
mean those who don't want them can just skip checking them out,
and we get the upstream revision tracking for free (since that's
how submodules work).

Cheers,

Jelmer