Samba CI mins now totalled: ~10million per month: Help needed

[Andrew Bartlett]

Posted publicly to samba-technical as nothing here is really private,
and perhaps our broader community can step up and help.

[feel free to follow up to the team list if you need to say something
private]

Just saw that on the public runners we use 10million mins per month per
GitLab's stats.  These stats have only just become available. 

That's a lot of mins. 

See attached and at 
https://gitlab.com/groups/samba-team/-/usage_quotas#shared-runners-usage-quota-tab

It is hard to follow exactly what GitLab.com's plans are for moving
from what must be rather un-sustainable for them, but there are clues:

https://gitlab.com/gitlab-org/gitlab/-/issues/243722#proposal makes it
clear our free run will end soon.

I'm sure we, as free users, bring value to their platform but that's a
lot of CPU, even just environmentally!

I think they plan to give public open source projects a 'cost factor'
of 0.008, and 50,000 mins free, so that would be billed at only an
extra $290 USD per month (((9.87 [million mins] x 1,000,000) * 0.008
[cost factor] - 50,000 [gold plan free limit]) / 1000 [mins per $10]) *
10 [price for 1000 mins].

However they seem to be looking to move the 0.008 to 0.5 so this will
bite:
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/77882

We should certainly subscribe to the gold plan urgently, but we really
need to trim down our CI also, reducing duplicate tests etc would be a
good start.

The bad news is that 10million mins per month is a massive scale above
what we can run in any commercial or charity cloud for reasonable money
- if evenly run over the month it would be around 10,000 USD per month
at Catalyst, much more at Rackspace.

(This doesn't include those mins we already run at Rackspace, costing
us around 600 USD per month).

Our CI has given us amazing advantages and allowed the team to go
further, review and maintain more without rework as we break the tree
or stomp on each others toes.  I personally feel like it adds around an
additional 'developer' worth of bandwidth. 


However we need to:

 - Subscribe to GitLab Gold for Open Source (they got rid of a silly
auto-renewal clause that caused trouble previously).  I'll take this up
with the PLC.
 - Talk to GitLab (and not just on the tickets) via their Open Source
liaison.  I'll do this. 

 - Get a working GitLab CI build at OSU OSL.  We need more cloud hosts
we can use and an Openstack cloud will be the least additional work.
 Uri did some fine work trying to get a rancher-machine binary that
would work with OpenStack's V3 keystone API and GitLab CI, but I don't
recall it working.  (This would also unblock Catalyst Cloud, they
dropped the V2 API recently). 

 - Trim down any jobs that do not add value.  
   - Test less distributions and less tests in the samba-o3 jobs.  
   - Remove more duplicate tests from source3/selftest/tests.py where I
blindly added tests of everything to ad_dc as well as nt4_dc.  Most are
also tested in source4/selftest/tests.py

 - Be more strategic in general

If anyone has the bandwidth to help this would be most appreciated!

Thanks,

Andrew Bartlett
-- 
Andrew Bartlett (he/him)       https://samba.org/~abartlet/
Samba Team Member (since 2001) https://samba.org
Samba Team Lead, Catalyst IT   https://catalyst.net.nz/services/samba

Samba Development and Support, Catalyst IT - Expert Open Source
Solutions
-------------- next part --------------
A non-text attachment was scrubbed...
Name: samba-team-ci-usage.png
Type: image/png
Size: 58088 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20220317/a2768c53/samba-team-ci-usage.png>