libctdb testing tree (do not pull! unfinished!)

[Rusty Russell]

OK, this is hot off my (still slightly ill!) fingertips:

	git://git.samba.org/data/git/rusty/ctdb.git libctdb

The idea (and much code!) is taken from nfsim.  See the commit messages below for references, but the basic idea is that we run simple scripts which describe operations, one per line, and what they expect the results to be.

In addition, we wrap all operations which can fail (eg. write, read, malloc, socket, connect) and fork at that point.  The child fails the op, and the parent succeeds.  The child won't pass the scripted test, but it should not crash or leak memory.  If it does, we report what the pattern of successes/failures was to get to that point for easy debugging.

For bonus points, run the entire thing under valgrind.  That catches use-after-free on error paths and other really nasty stuff.

I've only implemented "connect", "attachdb", "detachdb" and "disconnect" and it's already found a bug.  See below.

Cheers!
Rusty.

commit ceae5908f86348148d84b2b495dfa2fdbfa671c3
Author: Rusty Russell <rusty at rustcorp.com.au>
Date:   Thu Jun 17 17:10:59 2010 +0930

    libctdb: implement ctdb_disconnect and ctdb_detachdb
    
    These are important for testing, since we can easily tell if we
    leak memory if there are outstanding allocations after calling
    these.
    
    Signed-off-by: Rusty Russell <rusty at rustcorp.com.au>

commit 09faa7a375c5721424f4accc32a8bb09ce189a59
Author: Rusty Russell <rusty at rustcorp.com.au>
Date:   Thu Jun 17 17:36:38 2010 +0930

    libctdb: test infrastructure
    
    This introduces 'ctdb-test', a program for testing libctdb.  It takes
    commands on standard input (with reduced functionality) or an input file.
    
    It still needs some cleaning up, but you can uncover a bug in libctdb
    today simply by running a simple attachdb test:
    
    	$ ctdb-test tests/attachdb1.txt
    
    It will print out a crash, and the path of successful and failed
    operations which lead to it:
    
    	...
    	Child signalled 11 on failure path: [malloc]:1:S[socket]:1:S[connect]:1:S[malloc]:1:S[malloc]:1:S[malloc]:1:S[malloc]:4:S[malloc]:4:F
    
    Feed that failure path into ctdb-test using --failpath (under a debugger):
    
    	gdb --args ctdb-test tests/attachdb1.txt --failpath=[malloc]:1:S[socket]:1:S[connect]:1:S[malloc]:1:S[malloc]:1:S[malloc]:1:S[malloc]:4:S[malloc]:4:F
    
    And you hit the exact error.
    
    It is based on the fork-to-fail model of nfsim.  The relevant parts are
    from page 154 of the proceedings of 2005 Ottawa Linux Symposium Volume II:
    	http://www.linuxsymposium.org/2005/linuxsymposium_procv2.pdf
    
    Or our presentation of same (from slide 21):
    	http://ozlabs.org/~jk/projects/nfsim/nfsim.sxi
    
    Signed-off-by: Rusty Russell <rusty at rustcorp.com.au>