[Report]Automated testing of Samba with Windows Protocol Test Suites

[joeg at catalyst.net.nz]

Hi everyone,

Happy new year!

I've done some job to use Ansible and OpenStack Cloud to automate the
testing of Samba with Windows Protocol Test Suites
<https://github.com/Microsoft/WindowsProtocolTestSuites>.

So far, I've only tried KDC category, but the other ones are easy to
support.

This is the current KDC test result:

total: 85

inconclusive: 3

passed: 32

failed: 50

And this is the 50 failed tests:

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileCrossRealmTest.CanonicalizeSpnInReferralTgt
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileCrossRealmTest.UserInAnotherDomain
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.ComputerAccount_DES_PreAuthentication_Fail
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.DesDowngradeProtection
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.HwAuthentFlag
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.Krbtgt_DES_PreAuthentication_Fail
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.OkAsDelegateFlag
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UserAccount_DES_PreAuthentication_Fail
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UserAccountDisabled
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UserAccountExpired
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UserAccountLocked
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UserAccountPasswordMustChangePast
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UserAccountPasswordMustChangeZero
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UserDelegationNotAllowed
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UserPreAuthenticationNotRequired
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UsingPasswordWithDES
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorBadMatch
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorBadOptions
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorBadProtocolVersionNum
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorCannotPostdate
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorFieldTooLong
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorGeneric
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorMsgType
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorRepeat
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorTicketNeverValid
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorTktExpired
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorTktNotYetValid
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileFastTest.AdFxFastArmorInAuthenticator
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileFastTest.AdFxFastUsedInAuthenticator
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileFastTest.FastKrb_Error
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileFastTest.FastOptions_HideClientNames
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileFastTest.KrbFastFinishedAsRep
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileFastTest.KrbFastFinishedTgsRep
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileFastTest.PaFxFastAdvertise
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileFastTest.StrengthenKey
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileFastTest.UnsupportedFastArmorType
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileSiloTest.Protected_Users_Interactive_Logon_Computer_A2A2_Fail
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileSiloTest.Protected_Users_Interactive_Logon_Computer_A2A2_Succeed
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileTestSuite.InteractiveLogonUseExplicitFast
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileTestSuite.InteractiveLogonUseFast
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.PacTestSuite.APDS_KERBEROS_PAC_VALIDATION
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.PacTestSuite.ServerSignature
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_AdFxFastUsedInAuthenticator
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_FastKrb_Error
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_InteractiveLogonUseExplicitFast
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_InteractiveLogonUseFast
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_KrbFastFinishedAsRep
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_KrbFastFinishedTgsRep
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_StrengthenKey
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_UnsupportedFastArmorType


It's now quite easy to set up the testing network and run it, and we are
going to fix the failed ones gradually in future.

-- 
Joe Guo
joeg at catalyst.net.nz
Catalyst IT