too many layers in selftest

[Douglas Bagnall]

On 15/02/20 10:45 am, Andrew Bartlett via samba-technical wrote:

>> The way the test environments are set up and
>> invoked is still a mystery to me unfortunately :-(.
> 
> There are two many layers:
>  - GitLab CI
>  - The bootstrap docker image
>  - autobuild.py
>  - make test (selftest/wscript)
>  - selftest.pl
>  - selftest/targets/Samba*.pm
>  - selftest/*/tests.py

You are forgetting or subsuming format-subunit and filter-subunit.

Samba's test infrastructure is a psychedelic fractal -- if you
stare for too long at any part, intricate detail oozes out of the
cracks.

> We got to this because each layer was added without harming the layers
> below, because incremental change is how we do things in Samba.  But we
> have built quite the contraption!
> 
> It would involve a closer commitment to GitLab CI (which might not be
> popular this week), but long term we could bid autobuild.py a fond
> farewell and distribute the data it contains between .gitlab-ci.yml
> (which also does the task of running scripts in order) and selftest
> (which could be told about which environments run in which job). 

Apart from the ordinary works-for-us inertia, there is a tragedy of the
commons thing -- people will pay for new features, but not for basic
infrastructure (or UI, documentation, security).

Douglas