[clug] Tests
Brad Hards
bradh at frogmouth.net
Thu May 22 21:57:59 GMT 2008
On Thursday 22 May 2008 10:43:46 pm Sjors Provoost wrote:
> Tridge showed a brilliant piece of testing software after pizza time.
> I would definitely like to try something like that on the software I
> need to write the coming months. What is it and where can I find it?
It is "gentest", which is part of the samba torture suite.
http://git.samba.org/?p=samba.git;a=tree;f=source/torture;hb=v4-0-test
(Note: everything below this is potentially wrong - I had to work pretty hard
just to keep up with what Tridge was telling us).
Tridge was showing "gentest_smb2" which is the version of the tool for the
SMB2 protocol (or at least the file server parts that samba4 will implement -
not sure it is meant to cover the whole protocol). There is a version for the
original smb protocol too (just called "gentest"), and a special version for
oplock testing (called locktest). All of those are in the samba4 torture/
directory - see gitweb link above.
As I understand it, the test does semi-intelligent comparisons between two
servers (say A and B). The oplock version opens two connections to each
server so you can check locking results - not sure if the
gentest/gentest_smb2 versions do that yet.
The idea is that if you do the same operation on each server (A and B) - say a
CreateFile() type operation, and the results differ, then something is wrong.
Of course, that type of thing is pretty easy to find using some basic tests.
Now envisage a sequence of random-ish operations (A0...An, and B0...Bn).
Perform those on both servers, and you might get a difference at step n.
Since that could be really large, it doesn't make a very good test. However
of those n operations, some subset are probably irrelevant. What gentest does
is a bisection search (what did you expect from the guy with a Ph.D in
searching and sorting algorithms) that identifies a smaller set of operations
that actually replicate the problem. [I'm not sure it provably is
the "smallest set", because there theoretically could be multiple paths, but
that probably doesn't occur too much in reality.] There is a chance that the
server behaviour is not deterministic (e.g. if you replay the same sequence
of operations, it doesn't break in the same place, aka a bug), and that will
break the search.
The other part that makes it very smart is that it encapsulates a lot of
Tridge level knowledge about how the protocol works, and what is needed to
explore the API space. For example, when you ask Server A and Server B for
an operation that gives you back a file handle, you should expect that they
might be different (and that isn't an error) and that you'll need to pass the
right handle back to the right server for subsequent operations. Same for
results on files that pass back creation time (should be close, but won't be
identical).
The idea that you have two servers and compare results is pretty fundamental
to gentest and friends. I'm not sure how well that will apply to your
situation, but I wish you good luck!
Brad
More information about the linux
mailing list