working long-term with the MIT KRB5 codebase in the AD DC

Tue Nov 14 16:54:42 UTC 2017

Hi,

I tend to hope that with the next import of heimdal, we would have been
able to use a system Heimdal.

But the work I had to do in the last weeks shows to me that it's not
that easy :-(

As we don't have tests to show our KDC behaves exactly like a Windows
DC, we're so far away of being able to rely on a more or less fixed
system version of a KDC.

The actual problem was a customer with a complex web application
using IIS on Windows 2012, which makes use of S4U2Proxy in order to
communicate between frontend and backend servers.

It took a week comparing traces between Windows and Samba in order
to eleminate the differences in the generated Kerberos tickets
(including the PAC) in order to finally find that we hit the
timestamp mismatch between ticket and PAC problem again.

Kerberos might not be as complex as SMB1, but it's not far from that.
Which means we need real low level tests to prove we're implementing
things correctly, without relying on kerberos client libraries
or manual testing with Windows, we're basically where we were with SMB1
15 years ago.

Unless we have the prove that we're really compatible with a Windows
DC, we'll have to rely on having custom versions of the KDC code we use,
in order to provide a ready to use version of a KDC.

The reason why things like git submodule, which rely on downloading
code from a specific url, is a bad idea, see
https://git.samba.org/?p=metze/samba/wip.git;a=blob;f=submodule-problems.txt;h=48a2a63c9c9f28b4df09b3f73608ccbd6eda480b;hb=9842196d2de6e673d50f0749bf9734914456c66d
for the reasons why.

But in general build hosts typically don't have internet access
and wouldn't be able to download things automatically.

I'm fine with moving heimdal to third_party, after the next import
and then maintain our known good state in the lorikeet-heimdal repo
where we can rebase it more easily on heimdal master and try to
keep the patchset minimal (0 patches if possible).

In order to have a tested KDC with MIT I'd propose to have it also
under third_party, every other option is not useful for the near future,
until we can really rely on having a stable code base in the upstream
Kerberos projects (maybe only MIT in future).

The fact that we need to implement a lot of new Kerberos features soon,
in order to keep up with Windows 2012/2016 makes it even more problematic.

I hope to find the time to create the start for low level
kerberos tests, using pyasn1 and some python bindings for
the krb5_crypto_* apis. Then we're hopefully be able to
build something similar to the dcerpc raw protocol tests.

For the reference, here're the bug reports I've recently opened:

S4U2Proxy requests with encrypted authorization-data are rejected by a
Samba KDC
https://bugzilla.samba.org/show_bug.cgi?id=13131

The KDC on an RWDC doesn't send error replies in some situations··
https://bugzilla.samba.org/show_bug.cgi?id=13132

The content of the S4U_DELEGATION_INFO PAC element is filled wrong by a
Samba KDC
https://bugzilla.samba.org/show_bug.cgi?id=13133

Padding/alignment of PAC elements is done wrong on Samba KDCs
https://bugzilla.samba.org/show_bug.cgi?id=13134

The KDC logic arround msDs-supportedEncryptionTypes differs from Windows
https://bugzilla.samba.org/show_bug.cgi?id=13135

A Samba KDC doesn't include the RID of the primary group also in the rid
array of groups
https://bugzilla.samba.org/show_bug.cgi?id=13136

S4U2Proxy tickets from a Samba KDC don't pass PAC verification checks
(authtime mismatch)
https://bugzilla.samba.org/show_bug.cgi?id=13137

The attempts to fix this can be found here:
https://git.samba.org/?p=metze/samba/wip.git;a=shortlog;h=refs/heads/v4-7-s4u2proxy
https://git.samba.org/?p=metze/samba/wip.git;a=shortlog;h=refs/heads/v4-5-s4u2proxy

It will be a challenge to bring the fixes to our users
as it's currently completely untested code mostly.

metze

Am 28.10.2017 um 11:57 schrieb Andrew Bartlett via samba-technical:
> Thank you all for your contributions to this thread.  I think Michael
> covers the issue pretty well, so I'll follow up below:
> 
> On Mon, 2017-10-23 at 11:05 +0200, Michael Adam wrote:
>> On 2017-10-20 at 11:56 -0700, Jeremy Allison via samba-technical wrote:
>>> So there's a tension here between the needs of distro's and the needs
>>> of vendors (I'm considering Catalyst via Andrew as a specific Samba
>>> vendor here).
>>>
>>> Distros need Samba to work with the libraries shipped with the distro.
>>> An "upstream first" policy for all dependent libraries is a must.
>>>
>>> Vendors need flexibility to add new features to both libraries and
>>> Samba to work with them. That sometimes means prototyping features
>>> and after customer approval getting the changes pushed upstream.
>>>
>>> I don't want a fork of MIT. No one wants to fork important security
>>> libraries anymore. I don't want to keep a copy of Heimdal anymore. I
>>> think we can all agree on this :-).
>>>
>>> Moving both Heimdal and MIT to a git reference to an external tree is
>>> a good idea.
>>>
>>> My guess is the difficulty comes from this statement in Andrew's
>>> email:
>>>
>>>> What I propose is:
>>>> - Our build system uses a git reference (via a submodule or otherwise)
>>>> to check out and build MIT krb5
>>>> - In Samba master, this tracks either:
>>>>   - MIT master
>>>>   - a "
>>>> - In Samba release branches this tracks:
>>>>   - the release branch, the released version of MIT krb5 that we will
>>>> support
>>>> - This occur in parallel to the Heimdal build
>>>
>>> what we need to determine is what "Samba vendor fork of
>>> MIT in limited circumstances" actually means.
>>>
>>> So long as this *doesn't* mean "a copy of MIT in our tree on samba.org"
>>> I hope we can come to some compromise we can agree on.
>>
>> I didn't understand Andrew as suggesting to put a copy of
>> MIT into our tree. He was talking about a (git) reference.
>> And I think this is perfectly fine, as Andreas said, for
>> development purposes.
>>
>> So what problem are we trying to solve?
>> The problem that the version of MIT kerberos
>> available on build systems that the devs are using
>> (e.g. for autobuild) varies greatly and is frequently
>> not up to the needs of current Samba.
> 
> Yes, and that we need a way to automated this.  My team at Catalyst
> increasingly uses infrastructure as code (build hosts created
> dynamically in the cloud) and others use travis-ci via GitHub.  
> 
> (BTW this transition alone is also an incredible opportunity, and one I
> hope others can get on board with soon.)
> 
> Because fixing the install on one box won't help, we need a way to say
> 'use this version', and that in turn gives us a great advantage of
> being able to upgrade that version and use different versions for
> different builds (not a global one on sn-devel). 
> 
> It is important to be able to say that Samba 4.7 needs MIT version 1.9,
> but 4.8 needs 1.12 for example, and build against exactly those.  That
> way we don't accidentally backport use of a new feature without
> explicitly bumping the required version. 
> 
>> I see a point in abstracting our development
>> tree away from those dependencies by providing
>> a reference to a git tree of mit kerberos so that
>> we can build against this (at least optionally, if
>> the system does not provide sufficient krb support)
>> because krb is kind of special.
> 
> Indeed, that is very important.  We once were held back to only using
> Kerberos features supported in Solaris (for example), and with the PAC
> etc have been on the leading edge of Kerberos features.  In a long-term 
> where we finally ditch Heimdal, it is quite likely that our users will
> need a modern krb5 for Samba to build, even as a file server, and we
> should be able to provide it.  Naturally this happens even more often
> for a DC.
> 
>> I.e. upstream Samba could be made autonomous by this.
>> This is to my understanding exclusively for the
>> convenience of developers.
>>
>> Any downstreams (i.e. vendors and distros) should (or rather
>> would want to) solve this problem differently by providing
>> the libs with the needed support in their system.
>> It should imho also be the default to build against an
>> external (system) krb lib. And the mode of first fetching
>> and building a krb version should be exclusive to
>> "developer" builds and should need to be triggered explicitly.
>>
>> I also don't think the vanilla upstream should ever reference
>> any vendor for of krb5, but only official trees. Referencing
>> vendor krb5-forks would be done in vendor forks of Samba...
>>
>> Finally, I think once this is implemented, we should do the
>> same with heimdal if possible: i.e. remove the checked out
>> copy and replace it by a git reference that can optionally
>> be used in developer builds....
> 
> Yes, this is my hope.  It is a long road, but we need to get to a spot
> where developing new features, which may need to be done in tandem with
> upstream projects is equally confident, and where both are fully tested
> in autobuild.
> 
> Just a few thoughts...
>>
>> Michael
>>
>>
>>> I see some hope in Andreas's reply here:
>>>
>>>>> What I propose is:
>>>>>  - Our build system uses a git reference (via a submodule or otherwise)
>>>>> to check out and build MIT krb5
>>>>
>>>> That's fine just for development!
>>>>>  - In Samba master, this tracks either:
>>>>>    - MIT master
>>>>>    - a Samba vendor fork of MIT in limited circumstances
>>>>
>>>> That's extremly bad! An enterprise distribution will not allow a vendor fork
>>>> of MIT Kerberos. You use what is in the system or not.
>>>
>>> Can we get some consensus on what "is fine for development" means
>>> to both of you ?
>>>
>>> Andreas, how do you see Andrew being able to add needed features
>>> to AD+MIT to move our MIT implementation forward ?
>>>
>>> Andrew, how do you see being able to separate this out from
>>> master so the distros can keep a supported Samba running against
>>> the default shipped and supported crypto libraries ?
> 
> This is one of the most fraught parts of the challenge.  Suppose we do
> all the right thing, and add a new feature against either upstream
> master krb5, but then no release is made in time for our own releases. 
> 
> And even if the release is made, an enterprise distribution won't ship
> a new krb5 just for Samba, let along to support new Samba packages on
> old enterprise distributions like SerNet provides.  (Yet this is
> incredibly popular in our user communities). 
> 
> If we are ever going to make the transition to out-of-tree or even to
> just MIT, we need to crack this nut, provide something that an
> enterprise distribution can live with (pure system packages, but never
> update either krb5 or samba), our users can use (need newer, perhaps
> even patched krb5 with newer Samba), and our developers can develop
> against (need to do a trial autobuild against a proposed set of krb5
> patches). 
> 
> What we can't continue with is where the MIT build of the AD DC is not
> tested at all, where assuming a new version requires root@ to change
> sn-devel (let alone that this is not the only build platform in use),
> and where the model appears to be 'wait for a new MIT version to hit
> Fedora rawhide'. 
> 
> Thanks,
> 
> Andrew Bartlett
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20171114/cd7790b5/signature.sig>