Help: Error in samba4.local.talloc in autobuild

[simo]

On Fri, 2012-10-05 at 11:32 +0200, Jelmer Vernooij wrote:
> On Fri, 2012-10-05 at 00:55 -0400, simo wrote:
> > On Fri, 2012-10-05 at 14:10 +1000, Andrew Bartlett wrote:
> > > On Thu, 2012-10-04 at 23:57 -0400, simo wrote:
> > > > On Fri, 2012-10-05 at 13:49 +1000, Andrew Bartlett wrote:
> > > > > On Thu, 2012-10-04 at 23:46 -0400, simo wrote:
> > > > > > On Fri, 2012-10-05 at 12:11 +1000, Andrew Bartlett wrote:
> > > > > > > On Thu, 2012-10-04 at 22:07 -0400, simo wrote:
> > > > > > > > Hi,
> > > > > > > > I am trying to push my talloc_memlimit branch to master and I got this
> > > > > > > > error twice in autobuild that is absolutely criptic to me:
> > > > > > > > 
> > > > > > > > [1346/1554 in 1h7m46s] samba4.local.talloc
> > > > > > > > UNEXPECTED(error): samba4.local.talloc.memlimit (subunit.RemotedTestCase)(none)
> > > > > > > > REASON: _StringException: _StringException: was started but never finished!
> > > > > > > > UNEXPECTED(error): samba4.local.talloc.memlimit(none) (subunit.RemotedTestCase)
> > > > > > > > REASON: was started but never finished!
> > > > > > > > 
> > > > > > > > As far as I can tell samba4.local.talloc simply runs (again) the talloc
> > > > > > > > testsuite (why ??) and the normal tallo ctestsuite completes in a few
> > > > > > > > seconds. So I do not understand why it would timeout here.
> > > > > > > > 
> > > > > > > > stodut and error logs do not shed any light, can someone help me
> > > > > > > > understanding what is going on ?
> > > > > > > 
> > > > > > > What's happening here is that the test is failing, but the subunit
> > > > > > > stream is incomplete, that is a test is declared as starting, but not as
> > > > > > > finishing.  
> > > > > > > 
> > > > > > > However, the error code from the binary appears to be 0, as otherwise it
> > > > > > > would be giving the familiar 'missing torture_fail()' line instead.
> > > > > > > 
> > > > > > > So, the 'never finished' is not a time-out, but an error in formatting
> > > > > > > of the test stream. 
> > > > > > 
> > > > > > Ok 'which' test is failing ?
> > > > > > I have no feedback from autobuild here.
> > > > > 
> > > > > Well as you noticed, it says samba4.local.talloc.memlimit, so I would
> > > > > start there.
> > > > > 
> > > > > You may need to run it and make test manually and work that out - there
> > > > > is nothing autobuild special about this error, it's all from 'make
> > > > > test'.
> > > > 
> > > > Ok I've been looking again at the various talloc tests. So let me guess,
> > > > autobuild interprets strings spit to output that start with "test: name"
> > > > and then wait until it sees a string that start with "success: name" to
> > > > determine a test was successful or not.
> > > > 
> > > > Am I on something here ?
> > > 
> > > Yes, that's pretty much how subunit works. 
> > > 
> > > https://bazaar.launchpad.net/~subunit/subunit/trunk/view/head:/README#L108
> > 
> > Right, if only we had one central place where we document all these
> > special things (instead of READMEs in obscure subdirectories that appear
> > completely unrelated), or at least used a special macro or function in
> > the current tests instead of open_coding this protocol so that when
> > others look at tests they get a hint ...
> We do have convenience macros that are used by tests in Samba, but not
> in talloc in an attempt to keep it lightweight. 
> 
> selftest/README is the central document for this kind of stuff, and
> refers to the subunit README.
> 
> Suggestions for improvements welcome.

I didn't think of looking into selftest/ for a README, I looke in
various torture dirs and testsuite and couldn't find much.
Took me also a while to realize how scattered the tests where.

It would be really nice if we could move everything testing related into
a specific toplevel subdirectory, and not have it scattered all over,
and then have a README there that explains the basics of how to operate
the directory, what each subdir contains (pretty much like
selfetest/README does), what are best practices and conventions, etc...

It is very difficult right now, for someone that is starting or only
occasionally care about the testing framework to find where stuff is or
how to play with it.

Simo.

-- 
Simo Sorce
Samba Team GPL Compliance Officer <simo at samba.org>
Principal Software Engineer at Red Hat, Inc. <simo at redhat.com>