Microsoft tests with smbtorture

[Andrew Bartlett]

I've been looking over the proposed patches to test Samba against
Windows, and I want to propose a change in direction.

I've CC'ed samba-technical, because I think a wider audience here may
assist.

At the moment, you have proposed some small changes to Windows.pm and
the build farm upload scripts to run smbtorture tests against various
Windows servers, to validate that the tests are actually valid against
Windows.  The tests are configured to run against a set of Windows
servers at Microsoft, with external configuration by environment
variables for critical details that the test scripts need. 

This is much appreciated, but I fear it isn't a sustainable approach. 

Our experience with Windows testing has involved a number of parties, so
far always from outside the team, setting up various environments to
test smbtorture against Windows.  Over the period of operation, these
environments break, either because of the VM becoming corrupted (our
tests may not leave things clean) or because some aspect of Samba or the
build farm has changed, and the specialised windows testing has been
inadvertently broken.

I'm very keen to avoid this result this time, and I think it is vital
that the Samba Team 'own' the testing against Windows.  It would be
great to have Microsoft provide machines that we can test against, and
to maintain one copy of the environment, but if the overall process is
maintained 'outside' the team, then it will bitrot.

Much of this should have been said months ago, but then we had no better
suggestion, and the approach taken was reasonable.  Now we have a better
suggestion, and a process that the team owns and more particularly
*uses*.  That is, over the past few weeks we have developed 'wintest'.
This is a framework for testing Windows against Samba4 and Samba4
against Windows.  It is generic against multiple virtual machine hosting
systems (supporting VirtualBox, KVM and soon VMware ESX), and uses
snapshots to ensure a repeatable state. 

For information about wintest see http://blog.tridgell.net/?p=91

I think it's key that this effort be re-adjusted to be part of wintest.
We may then run the selftest.pl from within wintest, with the
configuration being passed in, or we may simply run the smbtorture
commands manually.  This isn't actually the important part.  What is
important is that everyone who runs wintest can run these tests and fix
the handlers.  This means we have a chance to ensure that the Samba team
can ensure that tests will keep passing, which is vital for the long
term success of this effort. 

(selftest.pl and even the build farm scripts are much less important
than they once were, as we now have the subunit tools and formatting
standard doing much of the heavy lifting).

Andrew Bartlett
-- 
Andrew Bartlett                                http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org
Samba Developer, Cisco Inc.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 190 bytes
Desc: This is a digitally signed message part
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20101202/7463cd37/attachment.pgp>