testsuite changes

[Jelmer Vernooij]

Hi Tridge,

On Fri, 2006-10-20 at 08:42 +1000, tridge at samba.org wrote:
>  > That's what I'm trying to figure out - hence why I am experimenting with
>  > running TORTURE-TALLOC as a subprocess. I don't really see the problem
>  > with having more binaries - they'll live under $LIBDIR/torture, not in
>  > $BINDIR. 
> That in itself creates problems! When I then run smbtorture will it
> run the one in the install dir or the build directory? Will I run one
> from an old tree, which version does it run? Does it mean smbtorture
> will not work if I haven't done a 'make install' ? Or will it have
> some search path, and thus have a difficult to predict behaviour?
Installed binaries use $libdir/torture, binaries built in the source dir
use $srcdir/bin/torture. If you run from an old tree, it'll use
$oldtree/bin/torture/. 

>  > The alternative is having a bunch of torture_* macros on top of
> testsuites that need to be able to run standalone
> 
> that's not the only alternative :)
> 
> As I see it, what you are trying to do is to capture output from these
> tests into your torture_ infrastructure. You are currently planning on
> doing that by capturing stdin/stdout on a child process. There are
> alternative methods that don't involve a child process :)
> 
> Here are some possibilities to consider:
> 
>  1) pass in a logging function to the tests. The logging function
>     would be similar to fprintf(), and you could even make it
>     fprintf() compatible. Test suites like the talloc one would call
>     this logging function. That will feed into the same parsing code
>     you have now (the parse code for child processes).
That makes it hard to run perl-based tests, for example, but I guess I
can live with it.

>  2) if you really want to get down and dirty, then dup2() file
>     descriptors 0 and 1 onto different fd numbers, then use a pipe for
>     fd 0 and 1, and capture the output internally. That means that
>     printf() inside the tests will redirect appropriately. Not
>     brilliantly clean, but quite possible. It's even doable on systems
>     that don't have dup2() (using the semantics of fd assignment
>     ordering).
All systems that Samba4 builds on have dup2() - we're relying on it now
to run LOCAL-TALLOC and that seems to be ok on all hosts.

However, this prevents output from being written to the screen
immediately. You'll have to wait until an entire testsuite finishes
before smbtorture can read from the fd and send output to the screen.

>  3) declare simple versions of the torture_ functions either as part
>     of libreplace or as part of a lib/testsuite/ library that is
>     linked into all tests, and included with talloc, libreplace etc.
I'd rather not have another dependency for talloc, replace, ldb, etc...
if we would like to allow test discovery (knowing which tests are going
to be run beforehand), then this code is non-trivial.

> I personally prefer (1), but I think all of the above would be much
> better than lots of little binaries and child processes.
I can live with 1.

>  > We already build two versions on smbtorture - one in $srcdir/bin
>  > that contains paths in $srcdir (TORTUREDIR is defined to be
>  > $SRCDIR/bin/torture, for example) ~/some/branch/bin/smbtorture
>  > works fine at the moment.
> what?? Why on earth are we doing that?
Because if you're using shared libraries then you need something like
that if you want to be able to run binaries from the source directory
without the need to install libraries to /usr/lib first or requiring the
user to export LD_LIBRARY_PATH.

The shared libraries are necessary for the openchange folks or others
who would like to link against our libraries. 

> This is smelling more and more like the dreaded libtool approach. When
> I test a binary in my source tree I want it to be absolutely identical
> to the one I install. I don't want the install procedure to mangle it.
The only thing that's different is the paths built in - those paths are
not the paths from the source directory but the ones set by configure -
sysconfdir, libdir, etc.

> Wow, I just did a 'make installbin' and I see it chews up an extra
> 250M in bin/install/ now. Yuck!
The alternative is forcing the user to export LD_LIBRARY_PATH correctly,
which is not portable and prone to errors.

> I know that games need to be played when using shared libs (which is
> one of the most idiotic design decisions ever made - shared libs
> should make life simpler not harder!!), but I am not using shared
> libs, so I don't want to have to play these silly games.
> 
> Can we please get rid of this?
Sure, but that means abandoning the quest for shared libs. If we're ever
to support them properly, we need to have them be built on developer
machines. Shared lib builds break easier than static ones, and the
shared lib support will just bitort.

If the current situation is already problematic, then I propose we
remove shared library support. There is no point in pursuing it as
things like that will be necessary.

>  > I'd rather say it's the unix way of doing things... having a lot of
>  > smaller binaries that each do one thing and do it right, rather than one
>  > monolithic application.
> oh, you've invoked the 'unix way' argument :-)
> 
> Have you run 'type echo' in bash lately? What about kernels - the
> 'modular unix way' would imply micro-kernels, and that didn't work out
> so well.
We still run make though, which in turn runs other programs - we don't
have an integrated development environment that loads the compiler,
linker, documentation viewer and who knows what into the same address
space. Even emacs isn't as bad.

I can see the point in integrating things that are related, but
smbtorture is basically a big switch() statement to run whatever test
the user specified.

Using binaries makes it very easy to add tests - adding a test is as
simple as dropping a binary into the torture directory. Tests can be
written in any language - and tools to do analysis on test results can
be written in 5 minutes.

> We already have _far_ too many binaries in Samba, and historically it
> has bitten us badly with all our different command line syntaxes for
> each binary (despite several efforts to unify them), and problems with
> running out of disk while building Samba (remember the several GB it
> takes on some platforms with -g?).
These aren't 'regular' binaries. They're not installed into bin/. The command line syntax is strictly enforced because they're being called by smbtorture
They can't confuse the user as they live in a private dir. The problem
with the big files was caused mainly by the fact that things like the
registry editor linked to printer libraries. TORTURE-TALLOC with -g on
my system is a mere 80k. 

>  > > What difference is there apart from the tiny main() function?
>  > We'd need wrapper macros/functions for the report functions 
>  > (torture_comment/torture_suite_add_suite/torture_suite_add_tcase/
>  >  torture_fail/etc)
> see above alternative solutions :-)
> 
> All you're doing now is using fork/exec as a way to capture stdout and
> stderr. That's a very heavy handed approach to intercepting IO !
The output of the individual test binary can be run through something
different than smbtorture though, such as a script that generates HTML
or stats.

The test binary doesn't have to be written in C and doesn't have to call
a specific function. This means it also works with shell scripts, python
scripts, perl, etc.

>  > It makes it impossible to run more than one test with smbtorture
>  > though as smbtorture will go down with the tests.
> If smbtorture crashes, I want it to crash! I don't want it to sail on
> after some part of the tests have seg faulted. This code that you are
> trying to isolate is linked into smbd, and we certainly don't want
> smbd to sail on when it seg faults. 
I'm not saying we should ignore the segfault, but if we have one test
segfault that shouldn't mean none of the other tests run.

Anyhow, to summarize:

 * There seems to be a lot of resistance against having separate
binaries. I still think this would be the best solution, but I don't
think I'll win this discussion, so let's have a tprintf() or something
as you proposed.

 * Is building separate binaries in bin/ and bin/install/ considered a
bad idea? Will building with shared libs enabled by default where
possible be a problem in the future? If so, I think there is no point in
pursuing building any of our libs as shared libraries.

Cheers,

Jelmer

-- 
Jelmer Vernooij <jelmer at samba.org> - http://samba.org/~jelmer/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://lists.samba.org/archive/samba-technical/attachments/20061020/a28cadf8/attachment.bin