testsuite changes

[Jelmer Vernooij]

Hi Tridge,

On Fri, Oct 20, 2006 at 02:54:56PM +1000, tridge at samba.org wrote:
> This should be easy to fix, just use pipe() on all platforms and use
> get the fd[0] and fd[1] ordering right, plus open 0 on /dev/null to
> give an empty fd for stdin. Or we can correctly emulate socketpair()
> using something like the socketpair_tcp() code that was used in the
> old torture code in Samba.
I'll have a look at backporting that.

>  > However, this prevents output from being written to the screen
>  > immediately. You'll have to wait until an entire testsuite finishes
>  > before smbtorture can read from the fd and send output to the screen.
> There are hackish games we could play to work around this, but they
> are not pretty. For example, we could have a timer event that reads
> from the pipe and writes to the real stdout, or we could use the same
> techniques as socket_wrapper to intercept a few calls. I think I
> prefer method (1), the logging function being passed in.
yeah, I think that one is the best one as well (apart from using a
subprocess, obviously ;-)

>  > >  3) declare simple versions of the torture_ functions either as part
>  > >     of libreplace or as part of a lib/testsuite/ library that is
>  > >     linked into all tests, and included with talloc, libreplace etc.
>  > I'd rather not have another dependency for talloc, replace, ldb, etc...
>  > if we would like to allow test discovery (knowing which tests are going
>  > to be run beforehand), then this code is non-trivial.
> the versions of these functions that are used in the standalone test
> suites (ie. when for example talloc is built standalone) would be very
> different and much simpler than the functions that smbtorture
> provides. Basically torture_comment() and torture_fail() would map to
> a printf() call. We could actually put these in libreplace. 

> Hmmm, I just had an idea ....

> We already have to support the BSD formatted error functions for
> heimdal. Those are the err(), errx(), warn() and warnx()
> functions. Try 'man err' on a Linux box for the man pages.

> What if we used those as the basis for our 'external' tests? So the
> talloc and libreplace test suites would call errx() and warnx() on
> errors and warnings appropriately. Then libreplace would provide
> replacements for those (currently they are supplied in
> heimdal_build/replace.c) but the libreplace versions would have a nice
> little hook that can be set to redirect the functions to the torture
> ui parse code.
I'd rather not do that - err() exits the program, thus skipping 
possible other tests and the standard output format isn't easily parsable. 
warn() and err() both also append strerror(errno) to the output and that's not
useful in a lot of cases.

I think using (1) (passing in a log function) instead makes much more
sense.

>  > Because if you're using shared libraries then you need something like
>  > that if you want to be able to run binaries from the source directory
>  > without the need to install libraries to /usr/lib first or requiring the
>  > user to export LD_LIBRARY_PATH.
> I don't mind supporting the use of shared libraries, and I like the
> idea of us providing shared libraries for other projects, but I don't
> like us having to pay the price in terms of disk space and complexity
> when we are building without shared libraries locally (which I think
> is a good default).

> When a user does choose to build with shared libraries, I don't think
> the LD_LIBRARY_PATH is such a problem. We can have a file 'shlibs.sh' in
> the source directory and users can do:

>   . shlibs.sh

> to set the right environment variables. This won't be needed if you
> haven't enabled shared libs in the build, but if you have then it
> allows us to avoid the double binaries.
That only works on systems with a ld.so that supports the LD_LIBRARY_PATH 
environment variable, so it would basically limit the shared library support 
to Linux and FreeBSD. I personally wouldn't have any problems with that,
in fact, I'd prefer it as it would make the build system a whole lot saner and 
we can't support shared libs on all systems anyway. But others might 
have different opinions...

>  > Sure, but that means abandoning the quest for shared libs. If we're ever
>  > to support them properly, we need to have them be built on developer
>  > machines. Shared lib builds break easier than static ones, and the
>  > shared lib support will just bitort.
> we can build shared libs on some of the build farm machines, so we'll
> get an email from the build system when it breaks. That will work as
> long as those emails aren't ignored :-)
It's still a far from ideal solution if it works that way. Build
failures that demonstrate themselves locally are much easier to solve
than trying to fix them and then waiting for the buildfarm to rebuild.

I don't mind though as long as I can build with shared libs locally
(which I'll certainly keep doing).

>  > I can see the point in integrating things that are related, but
>  > smbtorture is basically a big switch() statement to run whatever test
>  > the user specified.
> those tests are in fact strongly related - they are all testing
> components of Samba. They are also highly intermixed. The talloc test
> does things that particularly stress talloc, but just about every
> other test also tests talloc indirectly. The same is true of
> libreplace, ldb, tdb etc.
Sure, but that's still doesn't explain why it should be in the one big
binary. Just because make and gcc are related doesn't mean they have
to be one binary either :-)

>  > Using binaries makes it very easy to add tests - adding a test is as
>  > simple as dropping a binary into the torture directory. Tests can be
>  > written in any language - and tools to do analysis on test results can
>  > be written in 5 minutes.
> It was pretty easy to add LOCAL-TALLOC and LOCAL-LIBREPLACE using the
> old 'linked in' system as well :-)
Not really - it doesn't allow running tests unless they're linked into
smbtorture somehow.

>  > These aren't 'regular' binaries. They're not installed into
>  > bin/. The command line syntax is strictly enforced because they're
>  > being called by smbtorture
> except that to answer some of the problems I raised, the solution was
> to run them standalone :)
Well, that's only when running them inside of gdb.

>  > The output of the individual test binary can be run through something
>  > different than smbtorture though, such as a script that generates HTML
>  > or stats.
> ok, but I presume that will still work for the built-in tests? Or were
> you thinking of having a separate binary for all our RAW-*, RPC-* etc
> tests?
Yes, as long as we output subunit, that should be fine. 

The main reason I was doing this was to avoid duplicate code and extra
#ifdef _SAMBA_BUILD_'s in talloc or ldb. Ideally, I'd like Samba to
be buildable linked against a standalone install of ldb of talloc.
That way, the built-in and standalone builts are identical and it's
not possible to break one without breaking the other.

>  > The test binary doesn't have to be written in C and doesn't have to call
>  > a specific function. This means it also works with shell scripts, python
>  > scripts, perl, etc.
> yep, I think the piped_child approach is fine for shell scripts, perl
> and python scripts. I would like to keep it internal for js scripts
> and C code.
Ok, I'm fine with doing that for the C code. I don't care about JS :-)

>  > I'm not saying we should ignore the segfault, but if we have one test
>  > segfault that shouldn't mean none of the other tests run.
> Actually, that raises an interesting point. One thing that might be
> useful in the GUI is a "stop on error" button, so you can then attach
> gdb and see the error in more details (or just look at the most recent
> packets in wireshark). If the tests are a child process talking over a
> pipe, how would you stop on an error? :-)
Simply by killing it. There is no way to stop a test in the current
system either.

Cheers,

Jelmer
-- 
Jelmer Vernooij <jelmer at samba.org> - http://jelmer.vernstok.nl/
 14:39:06 up 9 days, 13:06,  4 users,  load average: 0.82, 0.46, 0.35
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 191 bytes
Desc: Digital signature
Url : http://lists.samba.org/archive/samba-technical/attachments/20061022/e26ad252/attachment.bin