smbtorture - why has it's output format changed?
Gary R. Schmidt
grschmidt at acm.org
Fri Nov 24 02:17:18 GMT 2006
Quoting Jelmer Vernooij <jelmer at samba.org>:
> We're adding more useful output to smbtorture, such as statistics.
> Another thing was trying to make sure that we can run with known
> failures, known skips, recognizing that just parts of tests failed
> allowing the output to be less verbose.
> If you are looking for machine-parseable output that is guaranteed
> stay the same, try the --format=subunit option.
Hmmm, it wraps the sub-tests, but I want to call each test by name.
> > I can understand adding tests, and I can almost understand changing
> > to allow sub-tests, but removing tests? No, you don't do that.
> I don't think we've actually removed any tests. Which ones are you
I understand that some of these may have been subsumed by other tests
(BASE-LOCK? appear to now be part of BASE-LOCK), but why should I (or
*anyone*) have to wade through the parameter list looking for possible
> Also, Samba 4 is still under heavy development. Things like this may
> will change over time. We don't make any guarantees.
Changes to the output of a test tool are to be expected as new tests are
added, but a wholesale change in format is just wrong.
Back in the dim and dark when I was taught how to do software, test
tools were designed first, and changes to output *format* was only
accepted after long and intense debate, as we also designed tools that
took that output and told us many useful things, such as whether the
total number of failures was increasing or decreasing.
Just for background, the reason I am using this is because we take the
near-to-latest Samba 3 source and build it for our own product, with a
couple of tweaks, which we then install over the top of any existing
Samba version. This means we need to do QA on it, and I am the bunny
who copped the job! (Note that it is still Samba, with all copyrights
maintained and so forth and we do push our modifications back into the
mainstream, we just can't wait for the the Linux distro's to incorporate
the changes we need.)
More information about the samba-technical