testsuite changes

[Jelmer Vernooij]

Tridge,

On Thu, 2006-10-19 at 13:59 +1000, tridge at samba.org wrote:
> A few more thoughts on this ...

>  > The output is in a format called `subunit' that is being standardized.
> I hope this doesn't mean you plan on having lots of our tests be
> standalone binaries! We already have far too many binaries in Samba,
> and having a pile of separate test binaries would be horrible I think :)
That's what I'm trying to figure out - hence why I am experimenting with
running TORTURE-TALLOC as a subprocess. I don't really see the problem
with having more binaries - they'll live under $LIBDIR/torture, not in
$BINDIR. 

The alternative is having a bunch of torture_* macros on top of
testsuites that need to be able to run standalone

> It would mean we'd have to install all those separate little tests as
> well, so that smbtorture can find them. Then we'd lose the ability to
> run ~/some/branch/bin/smbtorture to get that version of smbtorture.
We already build two versions on smbtorture - one in $srcdir/bin that contains paths in $srcdir (TORTUREDIR is defined to be $SRCDIR/bin/torture, for example)
~/some/branch/bin/smbtorture works fine at the moment.

> I was thinking of moving in the opposite direction, by allow our js
> tests to be run directly inside smbtorture, which just involves
> linking smbtorture to the js engine and abstracting out the core of
> scripting/ejs/smbscript.c as a ejs_run_script() function which could
> be called from smbtorture.

> I also wanted to move gentest, masktest, locktest etc into smbtorture.
That means smbtorture as one big monolithic app that depends on all
the rest of Samba... 

> I certainly like the abstractions you are putting in place for our
> test suites, but I don't like the separate binaries plan. It smells
> rather too much like libtool for my taste!
I'd rather say it's the unix way of doing things... having a lot of
smaller binaries that each do one thing and do it right, rather than one
monolithic application.

On Thu, 2006-10-19 at 13:49 +1000, tridge at samba.org wrote: 
>  > Using valgrind on ./bin/torture/LOCAL/TALLOC or using the
>  > --trace-children option to valgrind should fix that.
> yep, but that doesn't help on the build farm
Why? Can't we use it on the buildfarm?

>  > Alternatively, we could get smbtorture to wrap the commands it was
>  > executing in gdb or valgrind, though I'd prefer to avoid that if
>  > possible.
> I'd prefer to avoid that too
It's still possible though to run gdb on the individual binary.

>  > That would improve some things, but the advantage of using a subprocess
>  > is:

>  >  * identical code for the samba-build and the stand-alone build
> What difference is there apart from the tiny main() function?
We'd need wrapper macros/functions for the report functions 
(torture_comment/torture_suite_add_suite/torture_suite_add_tcase/
 torture_fail/etc)

>  >  * being able to detect segfaults and other errors (rather than test
>  > failures) in the testsuite
> This is the sort of thing I think is better detected with the code
> linked in, rather than a separate binary. The valgrind example is the
> most obvious example, where coding errors will show up in the build
> farm with it builtin, and not otherwise.
It makes it impossible to run more than one test with smbtorture
though as smbtorture will go down with the tests.

>  >  * being able to parse the output by something else than smbtorture. For
>  > example, we could have a script that parses the output and generates
>  > html or graphs from it.
> and that should be able to be run on the smbtorture output with the
> code builtin as well, no?
Yes, though it'd be rather ugly having one function call 

report("test: foo\n");

and then have that function parse it's arguments.

Cheers,

Jelmer
-- 
Jelmer Vernooij <jelmer at samba.org> - http://samba.org/~jelmer/