scan tests in make test (was Re: Retiring or fixing smbtorture?)

Volker Lendecke Volker.Lendecke at SerNet.DE
Wed Oct 24 06:05:52 MDT 2012


On Wed, Oct 24, 2012 at 03:57:39AM -0700, Richard Sharpe wrote:
> On Tue, Oct 23, 2012 at 6:17 PM, Andrew Bartlett <abartlet at samba.org> wrote:
> > On Tue, 2012-10-23 at 15:43 -0700, Richard Sharpe wrote:
> >> Hi Folks,
> >>
> >> Shouldn't we either retire smbtorture or fix it?
> >>
> >> For example, the TRANS2SCAN test fails when a call is made to
> >> cli_open/cli_openx (depending on version) to open a directory ("\\")
> >> but the underlying code sets the file attributes to not a directory.
> >>
> >> This is just bit rot. Should it be fixed? It could probably be simply
> >> fixed by adding a file_attributes parameter to the calls made all the
> >> way down.
> >
> > So, some of the history here is that we don't run scan tests in make
> > test, because they really need to be run against windows.  We then don't
> > run them against windows because we now have an official way of
> > discovering new protocol elements, rather than scanning and guessing.
> >
> > Part of the reason we don't run them in make test is that they tend to
> > be fairly slow, but because they don't validate their output they are
> > not particularly useful, except in filling logs with unexpected command
> > messages.
> >
> > The blocking of these is done by the selftest/skip file, and comments on
> > some other scan entries include:
> >
> > ^samba4.rpc.autoidl  # this one just generates a lot of noise, and is no
> > longer useful
> > ^samba4.rpc.countcalls # this is not useful now we have full IDL
> > ^samba4.rap.scan # same thing here - we have docs now
> > ^samba4..*trans2.scan # uses huge number of file descriptors
> > ^samba4.*.base.scan.ioctl # bad idea in make test
> > ^samba4.*.base.scan.pipe_number # bad idea in make test
> 
> OK, so, I think I understand what you are telling me :-)
> 
> However, QA tends to find it difficult to understand these things, and
> lots of people seem to still use smbtorture from source3 and then run
> into trouble with all the tests that fail for all sorts of reasons.

This is a well-known problem. There is a complete business
to be made around smbtorture-like Software. Look at the list
of companies at SDC offering SMB validation. I doubt that
any of those can be run without some level of manual
interpretation. Sure, smbtorture is really an acquired
taste, but I bet the competition is very good at selling you
the interpretation of the scan results.

Volker

-- 
SerNet GmbH, Bahnhofsallee 1b, 37081 Göttingen
phone: +49-551-370000-0, fax: +49-551-370000-9
AG Göttingen, HRB 2816, GF: Dr. Johannes Loxen
http://www.sernet.de, mailto:kontakt at sernet.de


More information about the samba-technical mailing list