[Report]Automated testing of Samba with Windows Protocol Test Suites

[joeg at catalyst.net.nz]

Hi Andreas,

I rebuilt samba with MIT Kerberos support (./configure.developer
--with-system-mitkrb5).

The new result is:

total: 85

inconclusive: 2

passed: 4

failed: 79

Failed tests:

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileCrossRealmTest.CanonicalizeSpnInReferralTgt

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileCrossRealmTest.UserInAnotherDomain

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.CaseInsensitive

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.ComputerAccount_DES_PreAuthentication_Fail

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.DesDowngradeProtection

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.ForwardableAndForwardedFlag

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.HwAuthentFlag

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.InitialAndPreAuthentFlag

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.Krbtgt_DES_PreAuthentication_Fail

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.MayPostdateAndPostDatedFlag

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.OkAsDelegateFlag

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.RenewableFlag

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UDPtoTCP

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UnknownPaType

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UserAccount_DES_PreAuthentication_Fail

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UserAccountDisabled

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UserAccountExpired

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UserAccountLocked

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UserAccountOutOfLogonHours

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UserAccountPasswordMustChangePast

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UserAccountPasswordMustChangeZero

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UserDelegationNotAllowed

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UserPreAuthenticationNotRequired

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UserTrustedForDelegation

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UserWithoutUPN

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UsingPasswordWithAES128

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UsingPasswordWithAES256

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UsingPasswordWithDES

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UsingPasswordWithRC4HMAC

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorBadIntegrity

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorBadMatch

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorBadOptions

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorBadProtocolVersionNum

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorCannotPostdate

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorClientPrincipalUnknown

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorETypeNoSupp

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorFieldTooLong

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorGeneric

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorMsgType

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorPreauthFailed

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorRepeat

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorServicePrincipalUnknown

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorSkew

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorTicketNeverValid

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorTktExpired

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorTktNotYetValid

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileFastTest.AdFxFastArmorInAuthenticator

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileFastTest.AdFxFastUsedInAuthenticator

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileFastTest.FastKrb_Error
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileFastTest.FastOptions_HideClientNames

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileFastTest.KrbFastFinishedAsRep

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileFastTest.KrbFastFinishedTgsRep

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileFastTest.PaFxFastAdvertise

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileFastTest.StrengthenKey
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileFastTest.UnsupportedFastArmorType

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileSiloTest.Protected_Users_Interactive_Logon_Computer_A2A2_Fail

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileSiloTest.Protected_Users_Interactive_Logon_Computer_A2A2_Succeed

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileTestSuite.InteractiveLogon

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileTestSuite.InteractiveLogonUseExplicitFast

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileTestSuite.InteractiveLogonUseFast

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.PacTestSuite.APDS_KERBEROS_PAC_VALIDATION

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.PacTestSuite.DomainLocalGroupMembershipWithDisableResourceSIDCompressionSet

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.PacTestSuite.DomainLocalGroupMembershipWithDisableResourceSIDCompressionUnset

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.PacTestSuite.KdcSignature
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.PacTestSuite.KERB_VALIDATION_INFO

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.PacTestSuite.PAC_CLIENT_INFO

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.PacTestSuite.PacMarkedAdIfRelevant

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.PacTestSuite.Request_no_PAC_TGT

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.PacTestSuite.Request_PAC_TGT

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.PacTestSuite.ServerSignature

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.PacTestSuite.UPN_DNS_INFO
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_AdFxFastUsedInAuthenticator

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_FastKrb_Error

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_InteractiveLogonUseExplicitFast

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_InteractiveLogonUseFast

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_KrbFastFinishedAsRep

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_KrbFastFinishedTgsRep

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_StrengthenKey

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_UnsupportedFastArmorType




On 03/01/18 20:18, Andreas Schneider wrote:
> On Tuesday, 2 January 2018 22:05:43 CET joeg--- via samba-technical wrote:
>> Hi everyone,
> Hi Joe,
>
>> Happy new year!
> same to you!
>
>> I've done some job to use Ansible and OpenStack Cloud to automate the
>> testing of Samba with Windows Protocol Test Suites
>> <https://github.com/Microsoft/WindowsProtocolTestSuites>.
>>
>> So far, I've only tried KDC category, but the other ones are easy to
>> support.
>>
>> This is the current KDC test result:
>>
>> total: 85
>>
>> inconclusive: 3
>>
>> passed: 32
>>
>> failed: 50
> Could please you also run this with MIT Kerberos?
>
>
> 	Andreas
>

-- 
Joe Guo
joeg at catalyst.net.nz
Catalyst IT