[Samba] Client/server test harness - Crucible 1.6

[Bryce Harrington]

Hi all,

At OLS last month I presented about doing automated client/server
testing of NFSv4.  In and after that talk there was some discussion with
Steve French about using the same framework for testing Samba, so I
thought it might be worthwhile to post about the framework on this list.
We've also just put out a new 1.6 release of Crucible; I've attached the
release notice below.  

The OLS presentation on NFSv4 testing is available here:

   http://developer.osdl.org/dev/nfsv4/site/documentation/OLS06.OSDL.v09.odp

however it is mainly presenting results and mentions the framework only
briefly; the talk I gave at OSCON about using Crucible with Xen is
probably a better overview:

    http://crucible.sourceforge.net/docs/oscon_2006.pdf

Briefly, Crucible is a collection of bash and perl tools (originally
inspired by the Samba build farm, in fact) for patching/booting kernels
on multiple machines and coordinating testing on them.  It uses a
network filesystem for communication between SUTs and the test driver,
and can be customized and extended with bash, perl, or other scripting
languages.  In the case that a machine's test kernel has failed, it is
able to get the console messages and reset the machine back to a
known-good kernel.  We've also added a network emulation layer (NetEm)
that allows for injecting network issues during test runs, so you can
see how the services respond as the network condition deteriorates.

Crucible will also automatically download and run tests on non-kernel
things, such as associated user-space libraries, tools, and so forth.
We're using Crucible to test Inkscape and Cairo, and of course testing
Crucible itself too.  ;-)

Steve French pointed out that most of the work done for automating the
testing of NFS can be reused for testing any other filesystem, so I
would love to explore how to use this for testing Samba.

Bryce


------------------------------------------------------------------------
Hi all,

With the positive responses from the OSCON talk about Crucible, we've
put together a new 1.6 release of it, available here:

    http://prdownloads.sourceforge.net/crucible/crucible-1.6.tar.gz

Crucible is a set of tools for automatically running tests on software
patches.  It is suited to doing tests of kernel code or of applications,
and is designed for doing network testing between multiple machines.
For more info, please see:

    http://crucible.sourceforge.net/

There's been a *bunch* of changes since 1.5 (mostly thanks to JasonN),
including:

   * Kerberos support for test plans

   * Incorporated Linux::Bootloader, etc. for building/booting kernel

   * Automatic update to EXTRAVERSION in kernel makefile

   * Support for build-only testing of non-kernel components

   * Bunch of cleanup and bug fixes

   * New features 'delete' and 'requeue' for the 'testrun' tool, for
     administrative control over problematic test runs.

   * New features 'lnimage', 'lsimage', 'mkimage', 'conlog', and 'power'
     to 'sut' tool.  These allow easy, uniform access to functionality
     in systemimager, conserver, and the remote power management tools.

   * Test plan commands have been converted into regular bash scripts.
     This makes customization of Crucible quite straightforward and
     easy.

   * Syntax of run_profile.txt files altered, so can now be sourced
     directly in scripts.  This eliminates a lot of the nutty sed|cut
     clutter that we'd been doing previously, and allows simplification
     of scripts.

Bryce