Buildfarm build_test script on an embedded device.

[Andrew Bartlett]

On Tue, 2011-10-04 at 22:24 -0500, Christopher R. Hertel wrote:
> Andrew Bartlett wrote:
> :
> > 
> > Chris,
> > 
> > I'm going to cut this off now, to save you spending any more time on
> > this.
> > 
> > I'm not going to accept this machine into the build farm.  There are
> > other ways we can help ensure that this class of device works well with
> > Samba, but the build farm isn't one of them.
> 
> No problem, if that's the best route to take.  There is a reason I asked
> whether it was a good idea when I did.  It's fine by me that we had to learn
> a few things before finding out that it was not a good fit.
> 
> > For a device to be a useful member of the build farm, it must:
> >  - return results in a reasonable period of time
> >  - not suffer serious resource constraints
> >  - be different enough to other platforms or a popular target.
> 
> I think this device meets the third of those criteria.  With regard to the
> first two, I believe that this class of device is very important for Samba,
> and that we should make a point of being able to support it.
> 
> > I think we should support devices this small, for small workloads, but
> > the Samba build process is not one of these, and the test process is
> > certainly not one of these.  
> 
> Agreed, yet the fact that we succeeded in running the full test suite (even
> though it took almost a full day the first time 'round) shows that this type
> of device can do a very good job for us, given the right environment.

Did it really succeed?  There are a number of fixed (in mins) timeouts
that should have fired well before 14 hours, so I'm sceptical.  Can I
get those logs?

> > To get better support for Samba on your dockstar, we should instead
> > ensure that an Arch Linux x64 machine is in the build farm, and if a
> > sufficiently fast and well-resourced host can be found, that an emulated
> > Arch Linux ARM machine is in the build farm.  The will help return
> > errors that developers can reproduce in a local VM (eg their python
> > insanity) and look at ARM portability issues.  
> 
> Why are you specifically targeting x64?

Or x86 - just something that doesn't need emulation on a modern CPU, so
we can just look for Arch-Linux oddity, and developers can set up a
local Arch Linux box to match the build farm when required. 

> Note that the Dockstar is an ARMv5 and that the ARMv5 is somewhat out of
> date.  There are multi-core ARM processors out now.  I will see if I can get
> hold of something else to try.

Getting something with a lot of ram will help a lot. 

> > Developers already face a minimum 1.5 hour wait to get changes into
> > master, and then the clock starts on all the build farm hosts.  The fast
> > hosts pick up the compile and test job, and return it in around 2 hours,
> > leaving at most 2 'try and fix $random_os' attempts per day.  If your
> > box takes 14 hours to run, then fixing it without local access is just
> > completely impractical.  
> 
> I agree in principle, but I would also suggest that this is a limitation of
> the way in which we use the build farm.  Don't get me wrong, I'm not
> suggesting that we change anything, particularly not to force an ARMv5
> system into the mix, just that I do believe that Samba should be able to run
> well on this sort of device and that it would be worth-while to make such a
> device available to the Team as a whole.

If you just want to make the box available, then that's just a matter of
accepting SSH logins.  The trouble is that the build farm is very much a
one-size-fits-all solution right now, and the resource demands have
escalated dramatically in the 10 years I've been involved with it. 

> > We have had slow or 'only once per day' machines in the build farm
> > before, and fixing them even for simple errors has proved incredibly
> > tedious.  
> 
> Are we fixing the machines or the build here?  Please explain.  If the build
> fails on such a platform, I claim it is still a bug.

There are a large number of failing boxes on the build farm.  These are
real bugs, but they also require very real amounts of developer time to
try and fix.  

The difficulty here is the same that was faced for all bugs before
autobuild - that the developer who sinks time into fixing the tests
typically is not the domain expert for the code that breaks it.

> > I do apologise for encouraging you to take this on, and for the time you
> > have spent.
> 
> Not at all.  The Build Farm is not my focus here.  My focus is ensuring that
> this sort of platform is well supported by Samba.  In that regard, the
> exercise has been a great success.
> 
> If the Build Farm is not the right place for this, perhaps we need to find
> another way to integrate it into our working environment.
> 
> Chris -)-----

If you can find a box that is around 10x more powerful (in ram and CPU)
that might help.  I'll know more when I see the logs.

Andrea Bartlett

-- 
Andrew Bartlett                                http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org