intermittent test failures in top level builds

[Andrew Bartlett]

On Fri, 2011-04-29 at 22:46 +0200, Stefan (metze) Metzmacher wrote:
> Am 29.04.2011 21:57, schrieb Jelmer Vernooij:
> > On Fri, 2011-04-29 at 21:29 +0200, Stefan (metze) Metzmacher wrote:
> >> Hi Tridge,
> >>
> >>>    http://git.samba.org/autobuild.flakey/2011-04-28-0842/samba4.stderr
> >>>    http://git.samba.org/autobuild.flakey/2011-04-28-0842/samba4.stdout
> >>
> >> Any idea why
> >>
> >> UNEXPECTED(error): samba4.nbt.winsreplication.owned
> >>
> >> is an error know instead of a failure, which would be ignored,
> >> because it's marked in knownfail?
> >>
> >> To me it seems that the knownfail feature doesn't work anymore.
> > knownfail only works for failures (i.e. we check for a particular error
> > but a different one was returned), not for errors (e.g. segfaults).
> > 
> > This is intentional; an error is often a sign of a broken testsuite.
> 
> I know, but that test used to generate a failure and not an error before.
> 
> So something seemed to be changed, because it started to be flakey again
> arround
> commit 1c5cc4a2d1449296a96e8a137dc177191df00c8d.
> 
> I'll change _NBT_ASSERT() and _NBT_ASSERT_STRING() to use the
> torture_assert macros,
> so that's a failure again, but we need to bisect why it is failing there
> now,
> while it didn't fail before with an error.

The big change that has happened in the past week is that we now build
with the s3 build and the change in 1c5cc4a is actually that with
f5487b3b4 we test with the s3 build in the top level.  

This has done two things.  The first, which I think you are seeing here
is that it there are more servers running and so timing-sensitive tests
may become more sensitive. 

The other thing that has shown up as intermittent failures come from the
51 duplicate symbols that remain.  A segfault in smbclient3 was caused
by the duplicate attrib_string() for example (the linker linked to the
'wrong' symbol), which is why I've made that common.  This kind of error
can be seen where a top level test of Samba3 fails in the top level
build, but not in the Samba3 autoconf or waf build. 

As such, I'll be working to merge or rename the remaining symbols in the
next week. 

The list of duplicate symbols can be seen by running. 

make SYMBOLCHECK=1  2>&1 | grep Symbol | cut -s -d ' ' -f 3| sort -u

Andrew Bartlett
-- 
Andrew Bartlett                                http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org