SOC Automated Windows Testing Project

[tridge at samba.org]

Jerry,

 > I'm confused a little so I'll go ahead an ask.  What is
 > the advantage of driving an smbtorture test against
 > a Windows server?  Or are you just setting up plumbing?
 > My conversations with Tridge led me to believe that
 > the goal would be to drive a Windows client to test a
 > Samba server.  Did I misunderstand?  If so, could you
 > fill me in (using short sentences)? :-)

The idea of this project is to do both samba->windows testing and
windows->samba testing. I'd consider the project successful if we get
both going.

The reason for samba->windows testing is that when we do only
samba->samba testing we occasionally hit situations where a change is
made that works fine samba->samba, but means we don't work at all
against windows. For example, we had some RPC changes a while back
that broke RPC against windows, and caused all our RPC-* smbtorture
tests against windows to fail, but didn't cause any failures against
samba. This is especially the case now that we share the code
generation for our client and server code - imagine if a change broke
the alignment rules in NDR. As the same code is used for both client
and server code, it won't show up in samba->samba testing, but would
show up in either samba->windows or windows->samba testing.

Although that example would work with windows->samba testing, the
problem is that we can only test a quite small proportion of the
protocol with windows->samba testing, as we don't have windows client
side tools to do wide protocol coverage. Using smbtorture we can test
a very large proportion of the protocol, but we need to validate these
tests against a windows server to ensure they really do something
useful.

We also need windows->samba testing of course, as that tests what our
users actually use Samba for! I'm hoping Brad will initially get a
system for automating windows->samba testing using batch files, then
if he has time, investigate the possibility of some sort of GUI
scripting.

For the samba->windows side of the project, I suggested to Brad that
he use RAW-QFILEINFO as a nice simple example initially. Once he has
that working really reliably (so no manual care of the setup needed)
then it should be easy to expand it to a much wider range of more
interesting tests.

I think the most challenging part of this project is getting that
reliability, so that once setup we don't need go and fiddle with the
server all the time. It should run for months/years, running the tests
on every commit for a select few hosts in the build farm (such as a
couple of my build farm machines that also run vmware, and have lots
of spare cpu cycles). If we have to login every few days to kick it
back into shape then it won't be useful, and making this stuff really
reliable will be tricky I think.

It also has to be reproducible. Brad needs to provide a simple
packaged way (docs, scripts, zip files etc) that allow anyone else to
quickly add windows testing to build farm machines.

On the more ambitious side, it would be nice to extend this later to
include other SMB/CIFS implemenattions, such as OS/2, Cascade, NetApp
etc. I'd really like to be able to look at the build farm at any time
and say "yep, we currently proposely support Apple, OS/2, WinXP, WinNT
and Win9X clients an servers for the following tasks" and not find out
when we do a release that we've broken something important.

Cheers, Tridge