working long-term with the MIT KRB5 codebase in the AD DC

[Andrew Bartlett]

Thank you all for your contributions to this thread.  I think Michael
covers the issue pretty well, so I'll follow up below:

On Mon, 2017-10-23 at 11:05 +0200, Michael Adam wrote:
> On 2017-10-20 at 11:56 -0700, Jeremy Allison via samba-technical wrote:
> > So there's a tension here between the needs of distro's and the needs
> > of vendors (I'm considering Catalyst via Andrew as a specific Samba
> > vendor here).
> > 
> > Distros need Samba to work with the libraries shipped with the distro.
> > An "upstream first" policy for all dependent libraries is a must.
> > 
> > Vendors need flexibility to add new features to both libraries and
> > Samba to work with them. That sometimes means prototyping features
> > and after customer approval getting the changes pushed upstream.
> > 
> > I don't want a fork of MIT. No one wants to fork important security
> > libraries anymore. I don't want to keep a copy of Heimdal anymore. I
> > think we can all agree on this :-).
> > 
> > Moving both Heimdal and MIT to a git reference to an external tree is
> > a good idea.
> > 
> > My guess is the difficulty comes from this statement in Andrew's
> > email:
> > 
> > > What I propose is:
> > > - Our build system uses a git reference (via a submodule or otherwise)
> > > to check out and build MIT krb5
> > > - In Samba master, this tracks either:
> > >   - MIT master
> > >   - a "
> > > - In Samba release branches this tracks:
> > >   - the release branch, the released version of MIT krb5 that we will
> > > support
> > > - This occur in parallel to the Heimdal build
> > 
> > what we need to determine is what "Samba vendor fork of
> > MIT in limited circumstances" actually means.
> > 
> > So long as this *doesn't* mean "a copy of MIT in our tree on samba.org"
> > I hope we can come to some compromise we can agree on.
> 
> I didn't understand Andrew as suggesting to put a copy of
> MIT into our tree. He was talking about a (git) reference.
> And I think this is perfectly fine, as Andreas said, for
> development purposes.
> 
> So what problem are we trying to solve?
> The problem that the version of MIT kerberos
> available on build systems that the devs are using
> (e.g. for autobuild) varies greatly and is frequently
> not up to the needs of current Samba.

Yes, and that we need a way to automated this.  My team at Catalyst
increasingly uses infrastructure as code (build hosts created
dynamically in the cloud) and others use travis-ci via GitHub.  

(BTW this transition alone is also an incredible opportunity, and one I
hope others can get on board with soon.)

Because fixing the install on one box won't help, we need a way to say
'use this version', and that in turn gives us a great advantage of
being able to upgrade that version and use different versions for
different builds (not a global one on sn-devel). 

It is important to be able to say that Samba 4.7 needs MIT version 1.9,
but 4.8 needs 1.12 for example, and build against exactly those.  That
way we don't accidentally backport use of a new feature without
explicitly bumping the required version. 

> I see a point in abstracting our development
> tree away from those dependencies by providing
> a reference to a git tree of mit kerberos so that
> we can build against this (at least optionally, if
> the system does not provide sufficient krb support)
> because krb is kind of special.

Indeed, that is very important.  We once were held back to only using
Kerberos features supported in Solaris (for example), and with the PAC
etc have been on the leading edge of Kerberos features.  In a long-term 
where we finally ditch Heimdal, it is quite likely that our users will
need a modern krb5 for Samba to build, even as a file server, and we
should be able to provide it.  Naturally this happens even more often
for a DC.

> I.e. upstream Samba could be made autonomous by this.
> This is to my understanding exclusively for the
> convenience of developers.
> 
> Any downstreams (i.e. vendors and distros) should (or rather
> would want to) solve this problem differently by providing
> the libs with the needed support in their system.
> It should imho also be the default to build against an
> external (system) krb lib. And the mode of first fetching
> and building a krb version should be exclusive to
> "developer" builds and should need to be triggered explicitly.
> 
> I also don't think the vanilla upstream should ever reference
> any vendor for of krb5, but only official trees. Referencing
> vendor krb5-forks would be done in vendor forks of Samba...
> 
> Finally, I think once this is implemented, we should do the
> same with heimdal if possible: i.e. remove the checked out
> copy and replace it by a git reference that can optionally
> be used in developer builds....

Yes, this is my hope.  It is a long road, but we need to get to a spot
where developing new features, which may need to be done in tandem with
upstream projects is equally confident, and where both are fully tested
in autobuild.

Just a few thoughts...
> 
> Michael
> 
> 
> > I see some hope in Andreas's reply here:
> > 
> > > > What I propose is:
> > > >  - Our build system uses a git reference (via a submodule or otherwise)
> > > > to check out and build MIT krb5
> > > 
> > > That's fine just for development!
> > > >  - In Samba master, this tracks either:
> > > >    - MIT master
> > > >    - a Samba vendor fork of MIT in limited circumstances
> > > 
> > > That's extremly bad! An enterprise distribution will not allow a vendor fork
> > > of MIT Kerberos. You use what is in the system or not.
> > 
> > Can we get some consensus on what "is fine for development" means
> > to both of you ?
> > 
> > Andreas, how do you see Andrew being able to add needed features
> > to AD+MIT to move our MIT implementation forward ?
> > 
> > Andrew, how do you see being able to separate this out from
> > master so the distros can keep a supported Samba running against
> > the default shipped and supported crypto libraries ?

This is one of the most fraught parts of the challenge.  Suppose we do
all the right thing, and add a new feature against either upstream
master krb5, but then no release is made in time for our own releases. 

And even if the release is made, an enterprise distribution won't ship
a new krb5 just for Samba, let along to support new Samba packages on
old enterprise distributions like SerNet provides.  (Yet this is
incredibly popular in our user communities). 

If we are ever going to make the transition to out-of-tree or even to
just MIT, we need to crack this nut, provide something that an
enterprise distribution can live with (pure system packages, but never
update either krb5 or samba), our users can use (need newer, perhaps
even patched krb5 with newer Samba), and our developers can develop
against (need to do a trial autobuild against a proposed set of krb5
patches). 

What we can't continue with is where the MIT build of the AD DC is not
tested at all, where assuming a new version requires root@ to change
sn-devel (let alone that this is not the only build platform in use),
and where the model appears to be 'wait for a new MIT version to hit
Fedora rawhide'. 

Thanks,

Andrew Bartlett

-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba