[Report]Automated testing of Samba with Windows Protocol Test Suites

joeg at catalyst.net.nz joeg at catalyst.net.nz
Wed Jan 3 20:33:05 UTC 2018 

Hi Andreas:

This is the result for Kerberos test suite:

total: 168

inconclusive: 5

passed: 37

failed: 126

And these are the failed ones:

Microsoft.Protocol.TestSuites.Kerberos.TestSuite.AZOD.AZODSyntheticTest.CBAC_Smb2_Negative_AccessFile
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.AZOD.AZODSyntheticTest.CBAC_Smb2_Possitive_AccessFile
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.AZOD.AZODSyntheticTest.CrossRealm_Smb2_Negative_AccessFile
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.AZOD.AZODSyntheticTest.CrossRealm_Smb2_Possitive_AccessFile
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.AZOD.AZODSyntheticTest.DAC_Smb2_Negative_AccessFile
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.AZOD.AZODSyntheticTest.DAC_Smb2_Possitive_AccessFile
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Claim.ClaimsTest.Kerberos_CrossRealm_ADSource_User_Only_Transform_From_Config
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Claim.ClaimsTest.Kerberos_CrossRealm_ADSource_User_Only_Transform_Use_AD_and_CTA
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Claim.ClaimsTest.Kerberos_SingleRealm_ADSource_User_Only
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileCrossRealmTest.CanonicalizeSpnInReferralTgt
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileCrossRealmTest.CrossRealm_KdcSignature
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileCrossRealmTest.CrossRealm_KERB_VALIDATION_INFO
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileCrossRealmTest.CrossRealm_OtherOrgSIDinPACFailure
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileCrossRealmTest.CrossRealm_OtherOrgSIDinPACSuccess
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileCrossRealmTest.CrossRealm_PAC_CLIENT_INFO
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileCrossRealmTest.CrossRealm_PAC_DEVICE_CLAIMS_INFO
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileCrossRealmTest.CrossRealm_PAC_DEVICE_INFO
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileCrossRealmTest.CrossRealm_ReferralTgs_NoOkAsDelegateFlag
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileCrossRealmTest.CrossRealm_ServerSignature
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileCrossRealmTest.CrossRealm_UPN_DNS_INFO
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileCrossRealmTest.ForwardableTicket_Http
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileCrossRealmTest.ForwardableTicket_Ldap
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileCrossRealmTest.ForwardableTicket_Smb2
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileCrossRealmTest.Pa_Svr_Referral_Info
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileCrossRealmTest.ReferralTicketIncludePacByDefault_Http
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileCrossRealmTest.ReferralTicketIncludePacByDefault_Ldap
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileCrossRealmTest.ReferralTicketIncludePacByDefault_Smb2
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileCrossRealmTest.ReferralTicketWithPac_Http
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileCrossRealmTest.ReferralTicketWithPac_Ldap
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileCrossRealmTest.ReferralTicketWithPac_Smb2
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileCrossRealmTest.ReferralTicketWithPacAndClaim_Http
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileCrossRealmTest.ReferralTicketWithPacAndClaim_Ldap
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileCrossRealmTest.ReferralTicketWithPacAndClaim_Smb2
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileCrossRealmTest.TransitedPolicyCheckedFlag
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileCrossRealmTest.UserInAnotherDomain
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileHttpApTest.ChannelBindingSuccess_Http
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileHttpApTest.DetectAuthenticatorModification_Http
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileHttpApTest.DetectTicketModification_Http
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileHttpApTest.RequestDeviceClaim_Http
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileHttpApTest.RequestUserClaim_Http
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileHttpApTest.RestrictedKrbHost_Http
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileHttpApTest.ServicePrincipalName_Http
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileHttpApTest.ServiceTicketWithoutPac_Http
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileHttpApTest.TokenRestrictionDifferentMachineId_Http
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileHttpApTest.TokenRestrictionSameMachineId_Http
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileHttpApTest.UsingFAST_Http
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.ComputerAccount_DES_PreAuthentication_Fail
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.DesDowngradeProtection
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.HwAuthentFlag
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.Krbtgt_DES_PreAuthentication_Fail
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.OkAsDelegateFlag
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UserAccount_DES_PreAuthentication_Fail
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UserAccountDisabled
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UserAccountExpired
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UserAccountLocked
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UserAccountPasswordMustChangePast
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UserAccountPasswordMustChangeZero
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UserDelegationNotAllowed
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UserPreAuthenticationNotRequired
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.UsingPasswordWithDES
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorBadMatch
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorBadOptions
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorBadProtocolVersionNum
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorCannotPostdate
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorFieldTooLong
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorGeneric
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorMsgType
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorRepeat
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorTicketNeverValid
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorTktExpired
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileKrbErrorTest.KrbErrorTktNotYetValid
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileSmb2ApTest.ChannelBindingSuccess_Smb2
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileSmb2ApTest.DetectAuthenticatorModification_Smb2
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileSmb2ApTest.DetectTicketModification_Smb2
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileSmb2ApTest.RequestDeviceClaim_Smb2
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileSmb2ApTest.RequestUserClaim_Smb2
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileSmb2ApTest.RestrictedKrbHost_Smb2
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileSmb2ApTest.ServicePrincipalName_Smb2
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileSmb2ApTest.ServiceTicketWithoutPac_Smb2
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileSmb2ApTest.TokenRestrictionDifferentMachineId_Smb2
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileSmb2ApTest.TokenRestrictionSameMachineId_Smb2
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileSmb2ApTest.UseDESOnly_Smb2
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileSmb2ApTest.UsingFAST_Smb2
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileFastTest.AdFxFastArmorInAuthenticator
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileFastTest.AdFxFastUsedInAuthenticator
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileFastTest.FastKrb_Error
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileFastTest.FastOptions_HideClientNames
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileFastTest.KrbFastFinishedAsRep
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileFastTest.KrbFastFinishedTgsRep
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileFastTest.PaFxFastAdvertise
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileFastTest.StrengthenKey
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileFastTest.UnsupportedFastArmorType
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileSiloTest.Protected_Users_Interactive_Logon_Computer_A2A2_Fail
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileSiloTest.Protected_Users_Interactive_Logon_Computer_A2A2_Succeed
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileSiloTest.Protected_Users_Network_Logon_Computer_A2A2_Succeed
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileTestSuite.CrossRealmGetReferralTGT
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileTestSuite.CrossRealmNetworkLogonSmb2
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileTestSuite.InteractiveLogonUseExplicitFast
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileTestSuite.InteractiveLogonUseFast
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileTestSuite.NetworkLogonClaimsSmb2
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileTestSuite.NetworkLogonCompoundIdentitySmb2
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileTestSuite.NetworkLogonHttp
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KileTestSuite.NetworkLogonSmb2
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.PacTestSuite.APDS_KERBEROS_PAC_VALIDATION
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.PacTestSuite.PAC_CLIENT_CLAIMS_INFO
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.PacTestSuite.PAC_DEVICE_CLAIMS_INFO
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.PacTestSuite.PAC_DEVICE_INFO
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.PacTestSuite.ServerSignature
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_AdFxFastUsedInAuthenticator
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_CrossRealm_PAC_DEVICE_CLAIMS_INFO
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_CrossRealm_PAC_DEVICE_INFO
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_FastKrb_Error
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_InteractiveLogonUseExplicitFast
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_InteractiveLogonUseFast
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_KrbFastFinishedAsRep
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_KrbFastFinishedTgsRep
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_NetworkLogonCompoundIdentitySmb2
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_PAC_CLIENT_CLAIMS_INFO
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_PAC_DEVICE_CLAIMS_INFO
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_PAC_DEVICE_INFO
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_RequestDeviceClaim_Http
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_RequestDeviceClaim_Smb2
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_StrengthenKey
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_UnsupportedFastArmorType
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_UsingFAST_Http
Microsoft.Protocol.TestSuites.Kerberos.TestSuite.Rc4EncryptTest.RC4_UsingFAST_Smb2


On 03/01/18 20:18, Andreas Schneider wrote:
> On Tuesday, 2 January 2018 22:05:43 CET joeg--- via samba-technical wrote:
>> Hi everyone,
> Hi Joe,
>
>> Happy new year!
> same to you!
>
>> I've done some job to use Ansible and OpenStack Cloud to automate the
>> testing of Samba with Windows Protocol Test Suites
>> <https://github.com/Microsoft/WindowsProtocolTestSuites>.
>>
>> So far, I've only tried KDC category, but the other ones are easy to
>> support.
>>
>> This is the current KDC test result:
>>
>> total: 85
>>
>> inconclusive: 3
>>
>> passed: 32
>>
>> failed: 50
> Could please you also run this with MIT Kerberos?
>
>
> 	Andreas
>

-- 
Joe Guo
joeg at catalyst.net.nz
Catalyst IT

More information about the samba-technical mailing list