Investigating CI options for Samba test branches

[Andrew Bartlett]

On Tue, 2015-05-05 at 20:34 +0200, Michael Adam wrote:
> Hi Andrew,
> 
> On 2015-05-04 at 15:28 +1200, Andrew Bartlett wrote:
> > Towards my proposal for faster release cycles, I've been looking into
> > how we can ensure developers use their own time better.  In particular,
> > I'm really keen to see a situation where folks push 'try' branches to a
> > review server, and even if that isn't how the patch is finally merged,
> > that an autobuild 'just happens' against that tree.
> > 
> > That way, it is much less likely that the reviewer will have to re-push
> > things due to an autobuild failure, as if they look at the patch over 2
> > hours after it was written, the patch would have been automatically
> > tested.
> 
> While I don't see how this would spare the reviewer to re-push
> things, I generally agree with the idea of forcing some
> build and smoke-test (if not a complete make test, etc even)
> before any reviewer starts to look at a patchset.

> There is a lot of potential for avoiding wasted work
> by catching things with such pre-review automatisms!

Exactly. 

What I was getting at is that baring flapping tests, the reviewer isn't
as likely to have the final autobuild, possibly with other changes fail.

> Speaking of this, I have recently worked a little with
> such a system that does this from the original review
> request up to the final push, namely the gluster development
> process. This is build around self-maintainted instances
> of gerrit for review and jenkins for CI.
> This is set up in such a way that a patch(set) that is submitted
> to gerrit first undergoes some tests that are performed by
> jenkins. And only if the tests pass, a reviewer starts looking
> at the change request. A patch needs verified ticks (for the
> tests) and enough review ticks before it can be merged into
> the upstream tree. This final merge/push does not happen
> automatically but has to be triggered by the reviewer.

The key to this process, and to what I was looking for, is that the
natural latency between a patch being written and looked at is a great
time for a computer to do the testing.  The reason we so often don't do
that testing right now is that waiting 2 hours to press 'send' on the
e-mail isn't satisfying nor practical. 

> While we have tried gerrit once and it was not so well received,
> and I know that generally some people have objections against
> gerrit and jenkins because they are written in java, my point
> is not so much about the actual components but that it gets
> really nice and useful if something like this is carried
> through from start to end! The experience of the gluster
> workflow is very positive for me, personally!

I would really like us to try something in this space again.  Part of
what fell apart last time was just due to the way we rebased the gerrit
master repo automatically.  Nothing has emerged out of the ashes of that
experiment, and so instead we continue with manual processes. 

That is why I've been looking into the testing end of things, because I
think a review server is only really helpful if paired with a testing
one, that is the whole workflow.

> Regarding the options that you have listed below,
> I am not quite certain what to do. There are so many
> influencing factors...
> I think for CI jenkins is the de-facto open-source
> standard at the moment, downside (for some): written in java.
> trevis-ci is hosted and mostly tied to github and not
> completely free. One can probably set it up oneself, too.

travis-ci isn't open source, as far as I can tell. 

> For the sake of flexibility and control, it would be good
> to have a system that is self-maintained. But that needs
> resources for administration. 

This appears to be the sticking point, and hence my comment about
wishing an organisation we trusted was willing to run this 'as a
service' on Free Software (which appears to be our requirement). 

> And we need machines (HW/VM)
> to schedule the compile/test jobs on.

Particularly if we can use some kind of auto-deployed images (so that we
can move them around as offers of free help come and go), if we want to
run them ourselves then this won't be hard to get.

> The main point still: Such a thing will really start to
> rock when it is integrated with a review tool and with
> the authoritative upstream repository.

Indeed, and I that to get there, we need to decide this is a goal worth
more than our current exact processes, and so be willing to be flexible
to obtain the greater benifit.  

In particular, it is pretty clear we will need to accept that web
applications do not have the same security properties as SSH, but that
we can mitigate the risks (such as a sentinel repo that supports only
pulling with fast-forward to detect history fraud and using X.509 certs
for authentication).

Andrew Bartlett

-- 
Andrew Bartlett
http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20150506/5b0e8955/attachment.pgp>