CI beyond sn-devel

[Alexander Bokovoy]

On la, 18 marras 2017, Andrew Bartlett via samba-technical wrote:
> On Fri, 2017-11-17 at 15:35 -0800, Jeremy Allison via samba-technical
> wrote:
> > On Sat, Nov 18, 2017 at 12:15:05AM +0100, Timur I. Bakeyev wrote:
> > > Hi, Jeremy!
> > > 
> > > On Mon, Nov 13, 2017 at 8:08 PM, Jeremy Allison via samba-technical <
> > > samba-technical at lists.samba.org> wrote:
> > > 
> > > 
> > >     But without auto-regression tests on FreeBSD triggered on
> > >     every code submission, we are going to get occasional
> > >     compile breakage, and I don't see a way around that
> > > 
> > > 
> > > Can I offer iXsystems Inc. CI and build systems for Samba project to have CI
> > > and coverage tests so bugs like this won't slip into the release code?
> > 
> > This requires a lot of coordination to work. Right now, Team members
> > sit at their local boxes and type:
> > 
> > git autobuild
> > 
> > in the tree they want to push to master. This psuhes to a defined
> > tree on sn-devel, and runs a set of scripts that make everything,
> > run the regression tests suites, and if everything passes do the
> > final migration of the code into the master git repo.
> > 
> > If anything fails there are web-pages you can look at to see
> > what test or part of the compile failed.
> > 
> > You'd need to duplicate that environment, and we'd need some
> > sort of a hook to ensure build and tests succeed on both
> > Linux and FreeBSD before a real commit.
> 
> I agree with the sentiment and goals, but I actually think we need to
> move beyond (wrap) autobuild.  It can't scale beyond the host it runs
> on, and we need to move to something less bespoke.  
> 
> Have a look at this: Homu integrates with github (which we don't use as
> a primary platform as it is not free software), but does a lot of what
> we need, specifically one-at-a-time rebase and test:
> 
> https://github.com/servo/homu
> 
> If we had something like that, hooked up to a gitlab and a cloud-hosted 
> CI system such as gitabl-ci then we can test beyond just one OS
> (indeed, one host), and reject commits that break FreeBSD, are
> incompatible with Windows or fail the WSPP testsuite. 

We have a similar model with freeIPA: https://github.com/freeipa/freeipa-pr-ci/
 - a master git repository is hosted at pagure.io

 - a commit hook updates github repository with force-push, making that
   one a mirror

 - any pull request proposed against the github tree either by an
   authorized person or having a label to run a test by an authorized
   person triggers a pull request CI to run tests

 - these tests run on handlers that may be set up everywhere else; in
   freeIPA case we have official PR CI running on a Red Hat internal
   servers and on travis-ci infrastructure

 - once PR CI runs completed, their status is marked in the github pull
   request

 - once pull request is ACKed by a human, the code from PR can be merged
   to the master tree on pagure.io by a human. We use some scripts to
   automate this process -- the script checks PR status, its PR CI
   status, whether it can be applied by fast forward to a selected git
   branch and also is able to open pull requests for backports
   automatically

github is basically used in this scheme just to ease the process to
provide pull requests for contributors and to visualize results of PR CI
integration. The whole story starts with a git push to a remote place
and marking this push as 'authorized for PR CI run', this is equivalent
to 'git autobuild' in the current Samba Team practice. One can easily
keep these models compatible because all it really requires to have is a
place where git remotes are pushed and a way to mark those pushes as
'authorized'. A PR CI service could be pushed with builds coming from
anywhere and 'git autobuild' can still be used as an entry point to push
the changes.

Finally, one major difference to current Samba Team practice is that
autobuilds with PR CI would be public before they complete as the trees
need to be copied to a test runner somehow. Right now the tree being
tested is kept private until it the test succeeded and merged. We also
have private autobuilds that never get merged and just run to see a
success or a failure. This is important for security releases
coordination. So any such PR CI system would need to account for a need
of being run from private pushes and not publishing logs of those until
allowed (or allow seeing them under some account privileges).

I have freeIPA PR CI running also against my personal freeIPA clone on
github. That PR CI is using my own freeIPA PR CI runner instance on my
desktop. So every time I push something and propose as a clone to
myself, I trigger the test runs. Here is an example:
https://github.com/abbra/freeipa/pull/5.  Scroll down to "All checks
passed" to see four different scenarious that were executed and links to
actual logs. The logs are actually pushed to some Fedora hosted server
which is used as a logs storage for each build.

> > would currently doubt it, as many of the tests (like the
> > Linux kernel oplock and change notify ones) require Linux
> > specific features. Also, some of the tevent tests require
> > epoll etc. etc.
> > 
> > This is doable, but not an easy change to make.
> 
> I think this is the best place to start.  Document the current failures
> in a new knownfail.d file as a start. 
> 
> In terms of resources, one think that iXsystems may be able to help
> with is target hosts for the cloud CI I mention above, or paying for
> some cloud time.  
-- 
/ Alexander Bokovoy