Rackspace runners rebuilt: Also note on differences between our runners and gitlab.com runners

[Andrew Bartlett]

Douglas correctly points out that 20.04 would be bad, it is Ubuntu
22.04.  
Andrew Bartlett
On Fri, 2024-05-10 at 11:25 +1200, Andrew Bartlett wrote:
> Just a note to say I've rebuilt the Rackspace runners.  Still on
> Ubuntu 20.04 due to availability, but rebuilt.
> It will means that if you have not rebased your branch on current
> master, it will now be 'stuck' waiting for runner tags that won't
> appear. 
> One challenge I will note is that GitLab.com is now running an 2 CPU
> 8GB runner, but our 'untagged' runner at Rackspace is only a 4GB
> runner.
> To upgrade to an 8GB runner at Rackspace is double the price, because
> it comes bundled with 4 CPUs.  
> In normal use we would only use untagged runners after our CI mins
> expire (currently we have plenty) or if there is an outage or
> throttling, but for security patches we use this 100% of the time.
>  So we have a choice:  Assume we still fit the 'untagged' jobs in the
> 4GB of ram, or pay twice as much and not waste time chasing
> incompatibilities during security work. 
> A security CI run would then cost $24, rather than $12.  
> BTW, if we don't make a change now and are in a pinch, the
> configuration of tags -> runners is now in the GitLab GUI, so I would
> like to document that we can pause the 4GB runner and select 'allow
> untagged jobs'.   
> Finally, given the way the new GitLab authentication token setup
> works, the next person to rebuild the runners will want to rework the
> scripts to have the 'blue/green' pattern used in dev-ops.  So instead
> of 'testing' and 'production', we would have 'testing, prod-green and
> prod-blue' with different authentication tokens, so we can pause the
> new/old runners again in case the prod deploy doesn't go to plan. 
> Andrew Bartlett