failing tests in the testsuite on debian

[Michael Tokarev]

15.11.2022 18:27, Rowland Penny via samba-technical wrote:
..
> Hi Michael, I think you may be over thinking this ;-)

Heh! :) It very well might be!

> It is my understanding that the tests are designed to test the entire Samba suite at compile time. It is this way to ensure that changes do not break 
> something else. Some tests are expected to succeed, while others are expected to fail, it is when a test does something that it isn't expected to do 
> that you get errors.

Yes, that's why samba have knownfails.  Which, btw, fail if it didn't fail ;)
(I mean, if a test marked as knownfail is NOT failing, it is a testsuite failure).

> You can run individual tests, but this is usually only done when altering an individual part of Samba, a python script for instance.

Yeah.  I was thinking about not individual tests, but about group of tests already
defined in samba, like the selftest/quick list.  Here's the comment from this file:

# This file contains regexes matching the tests that should be run
# when doing a "quicktest" - verifying whether the build is working
# rather than trying to see what exactly is broken.
#
# This should be as quick as possible but cover as much code as possible.

To me it looks like a good way to use in a distribution right after next samba
release is built, exactly to do this: to test if the packager didn't break
anything.

> So, when the Samba tarball gets to you, you can be fairly sure that Samba is correct. Any errors, after you compile Samba and create debs from it, are 
> likely to be packing errors, perhaps you need your own tests for that ?

Yeah. It must be a packaging error. I want to know where my error is, find
it and fix it.  I want to know why this thing works at the samba side,
but does not work at my side, where my side is wrong and how to make it
right.

Sure I can replicate samba test suite, at least some of it, -- like from
the "quick" list above. Wouldn't it be a waste of efforts since it is
already written and in use?

> I see no use in packing the tests (if this is what you are trying to do), they are really only of use at compile time.

This is exactly the compile time I'm talking about ;)

Not even *any* compile, but after I do something with the Debian package
when I'm not exactly 100% sure for the result and want a way to verify
some basic functionality.

Though I understand fairy well this is testing a separate build,
not the thing which has been just built "for production"

> This is my understanding of the tests, if it is wrong, I am sure someone will jump in and correct my errors.

It looks like your understanding is correct.

Thank you for the comments!

/mjt