too many layers in selftest

[Andrew Bartlett]

On Fri, 2020-02-14 at 13:02 -0800, Jeremy Allison via samba-technical
wrote:
> On Sat, Feb 15, 2020 at 09:57:58AM +1300, Andrew Bartlett wrote:
> > On Fri, 2020-02-14 at 12:50 -0800, Jeremy Allison wrote:
> > > keep getting (for example):
> > > 
> > > https://gitlab.com/samba-team/devel/samba/-/jobs/438181318
> > > 
> > > Failed to start up environment 'ad_member' at /tmp/samba-testbase/b21/samba-admem/selftest/target/Samba.pm line 125.
> > > 7206 samba can't start up known environment 'ad_member' at /tmp/samba-testbase/b21/samba-admem/selftest/selftest.pl line 840.
> > > 7207 ==> /builds/samba-team/devel/samba/samba-admem.stdout <==
> > > 7208 [1(0)/138 at 0s] samba.tests.pam_winbind(local+krb5)(ad_member)
> > > 7209 ERROR: Testsuite[samba.tests.pam_winbind(local+krb5)(ad_member)]
> > > 7210 REASON: unable to set up environment ad_member - exiting
> > > 
> > > The code changes I'm making (DFS/VFS stuff) aren't anything
> > > to do with this.
> > > 
> > > Any ideas on what might be going wrong ?
> > 
> > I'm pretty sure it is your changes.  This looks generic enough to
> > reproduce locally, so try that to get good logs. 
> > 
> > make test FAIL_IMMEDIATELY=1 TESTS='--include-env=ad_member --include-env=ad_member_idmap_rid --include-env=ad_member_idmap_ad --include-env=ad_member_rfc2307'
> > 
> > It does say this a lot earlier, which does look like DFS/VFS stuff:
> > 
> > checking for winbindd
> > wait for smbd
> > tree connect failed: NT_STATUS_CONNECTION_DISCONNECTED
> > tree connect failed: NT_STATUS_CONNECTION_DISCONNECTED
> > tree connect failed: NT_STATUS_CONNECTION_DISCONNECTED
> > 
> > Sorry,
> 
> Don't be sorry, that's really helpful info to help
> track stuff down - thanks ! :-).
> 
> The way the test environments are set up and
> invoked is still a mystery to me unfortunately :-(.

There are two many layers:
 - GitLab CI
 - The bootstrap docker image
 - autobuild.py
 - make test (selftest/wscript)
 - selftest.pl
 - selftest/targets/Samba*.pm
 - selftest/*/tests.py

We got to this because each layer was added without harming the layers
below, because incremental change is how we do things in Samba.  But we
have built quite the contraption!

It would involve a closer commitment to GitLab CI (which might not be
popular this week), but long term we could bid autobuild.py a fond
farewell and distribute the data it contains between .gitlab-ci.yml
(which also does the task of running scripts in order) and selftest
(which could be told about which environments run in which job). 

Andrew Bartlett

-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba