scan tests in make test (was Re: Retiring or fixing smbtorture?)

[Richard Sharpe]

On Tue, Oct 23, 2012 at 6:17 PM, Andrew Bartlett <abartlet at samba.org> wrote:
> On Tue, 2012-10-23 at 15:43 -0700, Richard Sharpe wrote:
>> Hi Folks,
>>
>> Shouldn't we either retire smbtorture or fix it?
>>
>> For example, the TRANS2SCAN test fails when a call is made to
>> cli_open/cli_openx (depending on version) to open a directory ("\\")
>> but the underlying code sets the file attributes to not a directory.
>>
>> This is just bit rot. Should it be fixed? It could probably be simply
>> fixed by adding a file_attributes parameter to the calls made all the
>> way down.
>
> So, some of the history here is that we don't run scan tests in make
> test, because they really need to be run against windows.  We then don't
> run them against windows because we now have an official way of
> discovering new protocol elements, rather than scanning and guessing.
>
> Part of the reason we don't run them in make test is that they tend to
> be fairly slow, but because they don't validate their output they are
> not particularly useful, except in filling logs with unexpected command
> messages.
>
> The blocking of these is done by the selftest/skip file, and comments on
> some other scan entries include:
>
> ^samba4.rpc.autoidl  # this one just generates a lot of noise, and is no
> longer useful
> ^samba4.rpc.countcalls # this is not useful now we have full IDL
> ^samba4.rap.scan # same thing here - we have docs now
> ^samba4..*trans2.scan # uses huge number of file descriptors
> ^samba4.*.base.scan.ioctl # bad idea in make test
> ^samba4.*.base.scan.pipe_number # bad idea in make test

OK, so, I think I understand what you are telling me :-)

However, QA tends to find it difficult to understand these things, and
lots of people seem to still use smbtorture from source3 and then run
into trouble with all the tests that fail for all sorts of reasons.

So, from that perspective I think we should either fix those parts
that fail because of bit-rot or simply remove them if the tests are
worthless.

However, I will have a look at the Samba 4 version of smbtorture and
see if I can specify a set of tests for QA that are reasonable based
on the exclusions list you have pointed out.

-- 
Regards,
Richard Sharpe
(何以解憂？唯有杜康。--曹操)