An even better GitLab CI (was: Re: Document GitLab as the only way to contribute to Samba?)

[Andrew Bartlett]

On Fri, 2019-06-28 at 14:40 +0200, Michael Adam wrote:
> On 2019-06-28 at 15:08 +0300, Uri Simchoni via samba-technical wrote:
> > On 6/28/19 2:46 PM, Michael Adam via samba-technical wrote:
> > > Ugh. That's really strange. Is this a gitlab design choice?
> > > Because if it is, it's finally a thing (and quite a major one),
> > > where gitlab is way worse than github. ;-)
> > > 
> > 
> > To be fair, I don't know if GitHub even has CI. What I've seen in  FLOSS
> > projects is that they have their Jenkins/Travis instance or something of
> > that sort (and who knows how many runners), and each PR would trigger a
> > run there, and success/failure would annotate the PR. GitHub just
> > provides the webhook.
> 
> That is true. CI is always external.
> But you typically configure the repo to require those CIs passing
> for a PR to become merge-able.
> 
> > I do agree with the bottom line though...
> 
> And don't get me wrong: I'm not advocating for github here.
> I was just comparing with something I know better.
> And if we could somehow tie the beefier CI with the core repo
> and run it on every MR, that would be great.
> But I probably need to read up on gitlab and it's ci etc.

When we used Travis CI on GitHub we did the same thing, except we never
had the full set of tests, it was always a cut down set.

It started as just a compile and very few tests, but enough to catch
the typical python whitespace errors and mixups between docs and
loadparm.  Just before we killed it we got it similar to the 'shared'
set on GitLab (I had to tweak it hard to get the run under an hour
total). 

I'm also kind of glad the CI is attached to the repo in one sense only:
The private runners are (after credits) at the Samba Team's cost. 
Abuse of free CI runners is a real thing, and so I'm more comfortable
with the risk when we can manually vet/ban users.

The fix is to:
 - Reduce the time spent in each job by:
  - use the pipeline feature in GitLab more to split the compile into a
different job
  - passing the compiled output (or primed ccache cache) to the 'test'
runners.  

 - Remove the dependency on an ext4 file system in our SMB tests or
find a way to get to ext4 on the shared runners (and not unionfs)

 - Reduce further the resource use of our tests
  - shut down smbd/samba processes when we are done with them in
selftest.pl
  - optimise the memory use of smbd/samba even more

All very much practical tasks, we just need help.  I'm always available
to consult, but I can't take on this much myself.  It would suit a
(new?) enthusiastic developer who is keen to make a mark and is willing
to streach the boundaries and try the 'impossible' to prove it isn't
really.

Thanks,

Andrew Bartlett
-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba