Rackspace runners rebuilt: Also note on differences between our runners and gitlab.com runners

[Andrew Bartlett]

Just a note to say I've rebuilt the Rackspace runners.  Still on Ubuntu
20.04 due to availability, but rebuilt.

It will means that if you have not rebased your branch on current
master, it will now be 'stuck' waiting for runner tags that won't
appear. 

One challenge I will note is that GitLab.com is now running an 2 CPU
8GB runner, but our 'untagged' runner at Rackspace is only a 4GB
runner.

To upgrade to an 8GB runner at Rackspace is double the price, because
it comes bundled with 4 CPUs.  

In normal use we would only use untagged runners after our CI mins
expire (currently we have plenty) or if there is an outage or
throttling, but for security patches we use this 100% of the time.  So
we have a choice:  Assume we still fit the 'untagged' jobs in the 4GB
of ram, or pay twice as much and not waste time chasing
incompatibilities during security work. 

A security CI run would then cost $24, rather than $12.  

BTW, if we don't make a change now and are in a pinch, the
configuration of tags -> runners is now in the GitLab GUI, so I would
like to document that we can pause the 4GB runner and select 'allow
untagged jobs'.   

Finally, given the way the new GitLab authentication token setup works,
the next person to rebuild the runners will want to rework the scripts
to have the 'blue/green' pattern used in dev-ops.  So instead of
'testing' and 'production', we would have 'testing, prod-green and
prod-blue' with different authentication tokens, so we can pause the
new/old runners again in case the prod deploy doesn't go to plan. 

Andrew Bartlett