CI beyond sn-devel

[Andrew Bartlett]

On Fri, 2017-11-17 at 15:35 -0800, Jeremy Allison via samba-technical
wrote:
> On Sat, Nov 18, 2017 at 12:15:05AM +0100, Timur I. Bakeyev wrote:
> > Hi, Jeremy!
> > 
> > On Mon, Nov 13, 2017 at 8:08 PM, Jeremy Allison via samba-technical <
> > samba-technical at lists.samba.org> wrote:
> > 
> > 
> >     But without auto-regression tests on FreeBSD triggered on
> >     every code submission, we are going to get occasional
> >     compile breakage, and I don't see a way around that
> > 
> > 
> > Can I offer iXsystems Inc. CI and build systems for Samba project to have CI
> > and coverage tests so bugs like this won't slip into the release code?
> 
> This requires a lot of coordination to work. Right now, Team members
> sit at their local boxes and type:
> 
> git autobuild
> 
> in the tree they want to push to master. This psuhes to a defined
> tree on sn-devel, and runs a set of scripts that make everything,
> run the regression tests suites, and if everything passes do the
> final migration of the code into the master git repo.
> 
> If anything fails there are web-pages you can look at to see
> what test or part of the compile failed.
> 
> You'd need to duplicate that environment, and we'd need some
> sort of a hook to ensure build and tests succeed on both
> Linux and FreeBSD before a real commit.

I agree with the sentiment and goals, but I actually think we need to
move beyond (wrap) autobuild.  It can't scale beyond the host it runs
on, and we need to move to something less bespoke.  

Have a look at this: Homu integrates with github (which we don't use as
a primary platform as it is not free software), but does a lot of what
we need, specifically one-at-a-time rebase and test:

https://github.com/servo/homu

If we had something like that, hooked up to a gitlab and a cloud-hosted 
CI system such as gitabl-ci then we can test beyond just one OS
(indeed, one host), and reject commits that break FreeBSD, are
incompatible with Windows or fail the WSPP testsuite. 

> Right now, can you type:
> 
> make test
> 
> on a FreeBSD Samba build and have all the tests pass ? I
> would currently doubt it, as many of the tests (like the
> Linux kernel oplock and change notify ones) require Linux
> specific features. Also, some of the tevent tests require
> epoll etc. etc.
> 
> This is doable, but not an easy change to make.

I think this is the best place to start.  Document the current failures
in a new knownfail.d file as a start. 

In terms of resources, one think that iXsystems may be able to help
with is target hosts for the cloud CI I mention above, or paying for
some cloud time.  

Andrew Bartlett

-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba