A plan

[Andrew Bartlett]

G'Day All.

I ran some tests overnight as promised.  

The first thing to say is that we (sadly) need to drop Douglas'
visualisation patches.  There are some python errors in the error cases
which show up only at the end of a full run (because the DB has junk in
it) that are not handled.

Then I think we need to run tests on less than this full branch.  

I'll try:
 - master plus the flapping additions
 - metze's branch minus Douglas' patches
 - asn's branch with the flapping additions (but not whoami)

We historically have always got into a muddle when we combine
everybody's patch into one push, it feels like it would save time but
actually takes longer:  This is because it assumes that all the patches
work, and for example I've put in good, tested code that failed, but
should have just failed its own autobuild, not held up yours. 

For master, I think some builds with just the flapping tests marked
would be good, then put that in.  Then do the rest by topic, owned by
the author.

In the medium term, Jamie (one of my new developers at Catalyst) is
working to untangle our testsuite inter-dependences.  The aim here is
to find sets of tests that:
 - are reliable
 - do no depend on each other
 - consume < 4GB of RAM
 - take less than 1 hour

(And then to split these into parallel test environments)

At Catalyst, running cloud builds for test is quite normal, often
before posting and generally before pushing.  But I've noticed that
even for me that the closer I get to the release deadline, the less
likely I am to wait for a full 5 hour build for the absolute final
patch.  I'm more likely to do what I did with the talloc patch: trust
earlier tests on different code and the newly written tests and aim at
autobuild.

What I would like to get to is a norm where when posting patches for
review, we post them to (say) gitlab by habit, and by the time they are
reviewed a clear 'passed/failed' flag is shown so we don't waste time
on patches that won't pass.  

In the meantime I'll run our 5-hour testsuite a few more times in hope
of getting the data on what can safely land for 4.8.

Thanks,

Andrew Bartlett
-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba