[clug] Tests

Brad Hards bradh at frogmouth.net
Thu May 22 21:57:59 GMT 2008


On Thursday 22 May 2008 10:43:46 pm Sjors Provoost wrote:
> Tridge showed a brilliant piece of testing software after pizza time.
> I would definitely like to try something like that on the software I
> need to write the coming months. What is it and where can I find it?

It is "gentest", which is part of the samba torture suite.
http://git.samba.org/?p=samba.git;a=tree;f=source/torture;hb=v4-0-test

(Note: everything below this is potentially wrong - I had to work pretty hard 
just to keep up with what Tridge was telling us).

Tridge was showing "gentest_smb2" which is the version of the tool for the 
SMB2 protocol (or at least the file server parts that samba4 will implement - 
not sure it is meant to cover the whole protocol). There is a version for the 
original smb protocol too (just called "gentest"), and a special version for 
oplock testing (called locktest). All of those are in the samba4 torture/ 
directory - see gitweb link above.

As I understand it, the test does semi-intelligent comparisons between two 
servers (say A and B). The oplock version opens two connections to each 
server so you can check locking results - not sure if the 
gentest/gentest_smb2 versions do that yet.  

The idea is that if you do the same operation on each server (A and B) - say a 
CreateFile() type operation, and the results differ, then something is wrong. 
Of course, that type of thing is pretty easy to find using some basic tests. 
Now envisage a sequence of random-ish operations (A0...An, and B0...Bn). 
Perform those on both servers, and you might get a difference at step n. 
Since that could be really large, it doesn't make a very good test. However 
of those n operations, some subset are probably irrelevant. What gentest does 
is a bisection search (what did you expect from the guy with a Ph.D in 
searching and sorting algorithms) that identifies a smaller set of operations 
that actually replicate the problem. [I'm not sure it provably is 
the "smallest set", because there theoretically could be multiple paths, but 
that probably doesn't occur too much in reality.] There is a chance that the 
server behaviour is not deterministic (e.g. if you replay the same sequence 
of operations, it doesn't break in the same place, aka a bug), and that will 
break the search.

The other part that makes it very smart is that it encapsulates a lot of 
Tridge level knowledge about how the protocol works, and what is needed to 
explore the API space.  For example, when you ask Server A and Server B for 
an operation that gives you back a file handle, you should expect that they 
might be different (and that isn't an error) and that you'll need to pass the 
right handle back to the right server for subsequent operations. Same for 
results on files that pass back creation time (should be close, but won't be 
identical).

The idea that you have two servers and compare results is pretty fundamental 
to gentest and friends. I'm not sure how well that will apply to your 
situation, but I wish you good luck!

Brad


More information about the linux mailing list