SOC Automated Windows Testing Project

[Brad Henry]

tridge at samba.org wrote:

>I think the main decision you need to make is whether you will 'drive'
>the testing from the unix side (the side running 'make test' in Samba)
>or from the windows side.
>
>The little python listener you describe is an example of driving the
>testing from the windows side, as most of the logic of the tests would
>be written using scripts that would execute on windows.
>
>The other approach would be to assume the windows box has some sort of
>remote terminal capability (eg. telnet server, ssh server or similar),
>and then drive that via an expect script (or similar) from the unix
>side.
>
>This has the advantage that all the testing would be driven by the
>machine running the 'make test'. So somewhere in our tests scripts
>we'd have something like:
>
> SETUP_TESTING_LINK
> SEND "net use z: \\server\share"
> EXPECT "The command completed successfully"
>
>(I'm just making up some pseudo-code syntax above - real expect
>scripting would look a bit different)
>
>  
>
I think that driving the tests from the unix side makes the most sense 
generally. I think that would give us an environment easier to use and 
maintain in the long run, because we wouldn't need to have any artifacts 
on the windows side, except for the remote access service.

I've been playing with copSSH (packaged OpenSSH in Cygwin - 
http://itefix.no/copssh). Assuming an SSH server would be nice, allowing 
us to scp file transfer and interact with the windows command line. I 
think that we'll find that it limits us to platforms NT and higher though.

>I think both approaches have their advantages. The main advantage I
>see of the expect approach is that it might require less stuff to be
>initially installed/maintained on the windows box (so no win32-python
>for example). The advantage of the approach you've suggested is it
>might be easier to do better error checking.
>  
>
I think that we'll likely find that we'll need something of a mix of 
both approaches. In cases where we want better error checking or 
functionality not directly offered through command line applications, we 
can copy windows scripts using scp and execute them over ssh, using 
expect for the cases where we need to interact with the tools that are 
available.

For example, joining a windows dc to a domain would be easy to do from 
the windows shell by copying/generating an answer file on the windows 
host and executing dcpromo /answer:<filename>.

>Why did you choose to use ADS calls rather than calling to the command
>line tools? I was imagining you'd do things like:
>
>  net user USERNAME PASSWORD /ADD
>  net user USERNAME /DELETE
>  net share test=c:\test /remark:"test share"
>
>Did you choose the ADS calls to get better error checking? That would
>certainly be a valid reason, but it would also make it harder to run
>these tests on Win9X, WinNT4 and other OSes that don't have ADS.
>
>Or maybe you need more control over the properties of the users/shares
>than you can get with net.exe commands?
>  
>
I decided to go with the visual basic scripts because i've tried to do 
windows admin scripts with batch files before, and found that there were 
cases where I was limited by the available cmd line applications, and 
would have wanted to interact with some of the api's directly. I can't 
remember the specifics at the moment, but I decided that I should get 
more comfortable with this approach so that I could fall back on it if I 
had to, without having to start from a vague idea :) I would rather go 
with the simplest solution where possible, I think the more custom code 
we run on the windows side, the more we'll have to worry about 
particularities between windows versions.

We can also use windows scripting to access an NT4-style domain, or 
local accounts. I prefixed the scripts with ads_ to distinguish them 
from possible future samdb_ or local_ versions. I think that windows 
scripting is available across windows 98 and higher (with the right ms 
updates) but I haven't tested it and so don't know for sure. These 
scripts mark my first actual output with windows scripting or visual basic.

> > 1. Samba test connects to the listening windows host and passes an ascii 
> > string containing the name of the test and relevant parameters.
> > 2. Windows host creates a user account and a share as requested. That 
> > user is configured on the windows host to have full control of that share.
> > 3. Windows host replys over the socket with an error code to indicate 
> > success or failure.
> > 4. Samba test runs "smbtorture -W TESTDOMAIN -U testuser%password 
> > //dc0.test.domain/testshare RAW-QFILEINFO"
> > 5. Samba test replies to windows host and windows host removes user, 
> > share and directory.
> > 6. Windows host replies success or failure of (5).
>
>yep, the trick there will be to recover from situations where the user
>already exists, or the share exists, or the password is rejected
>etc. You could either cope with this by assuming the test always
>starts off with a clean setup (due to a vmware snapshot restore for
>example) or you could catch the errors and fix them (eg. delete old
>user if user already exists etc).
>  
>
I think that it would be easier to just assume a clean snapshot, but I 
would like to recover, report the error and try to clean up (delete the 
existing share and recreate, delete the old user and recreate, etc). In 
cases where attempted recovery fails, i'm not sure if we should rebuild 
from the snapshot and try the test again, or report an error and quit. 
My feeling is that if recovery fails, we should rebuild from the 
snapshot and try the test again, but there may be cases i'm not thinking 
of where we wouldn't want to do that.

>
> > 7. Samba test creates a directory, and a user account, and a smb.conf 
> > which shares that directory, giving that user full control.
> > 8. Samba test replies to Windows host with the test name, username, 
> > password and share name.
> > 9. Windows host runs equivalent RAW-QFILEINFO test vs smbd and replies 
> > with its results to Samba test.
> > 10. Disconnect. Samba test halts smbd and removes directory and smb.conf 
> > file.
>
>for this half of the project (the windows->samba half) I think we will
>be mostly testing existing windows commands, like net, xcopy, del,
>cacls etc, rather than having anything close to RAW-QFILEINFO. We
>could port smbtorture to windows (its been done before), but I don't
>think it would win us much, as it would use our SMB library code, and
>what we really want to test is the Windows native SMB library code
>against a Samba server.
>  
>
That sounds good. I think that some of the tools from the windows 2000 
resource kit could be pretty handy in this regard. I haven't seen the 
2k3 res kit but i'm sure it would be similar.

>We could try for something like ifstest, and certainly running
>automated ifstest would be fantastic, but lets try for the simpler
>stuff first (if you haven't heard of ifstest let me know and I can
>give you some background on that).
>  
>
I've never heard of ifstest. Do you have a link?

>Alternatively, you could use interfaces between things like python and
>win32. It would be important not to use things like cygwin, as those
>tend to make quite different types of calls to what native win32 apps
>use. 
>
>Cheers, Tridge
>  
>
Thanks!
Brad