smbtorture - why has it's output format changed?

[Gary R. Schmidt]

Hi Jelmer,

Quoting Jelmer Vernooij <jelmer at samba.org>:

> We're adding more useful output to smbtorture, such as statistics.
> Another thing was trying to make sure that we can run with known
> failures, known skips, recognizing that just parts of tests failed
> and
> allowing the output to be less verbose.
Okay.

> If you are looking for machine-parseable output that is guaranteed
> to
> stay the same, try the --format=subunit option.
Hmmm, it wraps the sub-tests, but I want to call each test by name.

> > I can understand adding tests, and I can almost understand changing
> it 
> > to allow sub-tests, but removing tests?  No, you don't do that.
> I don't think we've actually removed any tests. Which ones are you
> missing?
Missing tests:
BASE-LOCK1
BASE-LOCK2
BASE-LOCK3
BASE-LOCK4
BASE-LOCK5
BASE-LOCK6
BASE-LOCK7
BENCH-CLDAP
BENCH-NBT
BENCH-OPLOCK
BENCH-READWRITE
BENCH-RPC
BENCH-TORTURE
BENCH-WINS
NBT-BROWSE
NBT-WINSREPLICATION-SIMPLE
RPC-DFS
SCAN-ALIASES
SCAN-CASETABLE
SCAN-EAMAX
SCAN-IOCTL
SCAN-MAXFID
SCAN-NTTRANS
SCAN-PIPE_NUMBER
SCAN-RAP
SCAN-SMB
SCAN-TRANS2
SCAN-UTABLE

I understand that some of these may have been subsumed by other tests
(BASE-LOCK? appear to now be part of BASE-LOCK), but why should I (or
*anyone*) have to wade through the parameter list looking for possible
replacements?

> Also, Samba 4 is still under heavy development. Things like this may
> and
> will change over time. We don't make any guarantees.
Changes to the output of a test tool are to be expected as new tests are
added, but a wholesale change in format is just wrong.
Back in the dim and dark when I was taught how to do software, test
tools were designed first, and changes to output *format* was only
accepted after long and intense debate, as we also designed tools that
took that output and told us many useful things, such as whether the
total number of failures was increasing or decreasing.

Just for background, the reason I am using this is because we take the
near-to-latest Samba 3 source and build it for our own product, with a
couple of tweaks, which we then install over the top of any existing
Samba version.  This means we need to do QA on it, and I am the bunny
who copped the job!  (Note that it is still Samba, with all copyrights
maintained and so forth and we do push our modifications back into the
mainstream, we just can't wait for the the Linux distro's to incorporate
the changes we need.)

        Cheers,
                Gary    B-)