Samba and Python2
Tim Beale
timbeale at catalyst.net.nz
Mon Feb 18 22:26:19 UTC 2019
Hi Metze,
The problem I hit was a lot of build/python interdependencies when
trying to use subunit on a build with --disable-python configured. I've
attached the actual errors I was hitting and my attempts to work through
the problems.
Aside from the dependency issues, a couple of things didn't feel quite
right with this approach:
1. It seemed to be working at cross-purposes - to both produce a
cut-down build with python excluded, and then to include a bunch of
python code just to test the build was successful.
2. In the medium-term, it seemed like a lot more python2 code to
maintain compatibility with (although granted, the approach with
no-python-tests.txt also has its own maintenance overhead).
I think ideally what we want here would be to separate out the compile
and test steps in CI, e.g. do make/install samba in a separate step
(i.e. use a separate repo or container volume), and then just point the
full selftest/autobuild (py3) scripts at the installed code. However, I
don't think it'd be particularly easy to do this currently.
Cheers,
Tim
On 18/02/19 10:24 PM, Stefan Metzmacher wrote:
> Hi Tim,
>
>> So here's something that runs some basic tests for the nopython build.
>> Basically I just bypassed all the subunit-filter python code and call
>> the Perl code directly. We then pass a manually edited test-list to
>> selftest.pl, which runs a single .sh test against each s3 testenv (apart
>> from the ad_member ones).
> What is the problem with the subunit code, shouldn't that work with any
> python version?
>
> metze
>
>
-------------- next part --------------
The first problem is fairly trivial - we can't find subunit because we didn't build/install it.
make test TESTS=fileserver
PYTHONHASHSEED=1 WAF_MAKE=1 ./buildtools/bin/waf test
'test' finished successfully (0.008s)
Waf: Entering directory `/home/timbeale/code/samba/bin/default'
Selected embedded Heimdal build
Waf: Leaving directory `/home/timbeale/code/samba/bin/default'
'build' finished successfully (25.605s)
test: running (/usr/bin/perl /home/timbeale/code/samba/selftest/selftest.pl --target=samba3 --prefix=./st --srcdir=/home/timbeale/code/samba --exclude=/home/timbeale/code/samba/selftest/skip --testlist="/usr/bin/python3 /home/timbeale/code/samba/selftest/tests.py|" --testlist="/usr/bin/python3 /home/timbeale/code/samba/source3/selftest/tests.py|" --testlist="/usr/bin/python3 /home/timbeale/code/samba/source4/selftest/tests.py|" --exclude=/home/timbeale/code/samba/selftest/slow --nss_wrapper_so_path=/home/timbeale/code/samba/bin/default/third_party/nss_wrapper/libnss-wrapper.so --resolv_wrapper_so_path=/home/timbeale/code/samba/bin/default/third_party/resolv_wrapper/libresolv-wrapper.so --socket_wrapper_so_path=/home/timbeale/code/samba/bin/default/third_party/socket_wrapper/libsocket-wrapper.so --uid_wrapper_so_path=/home/timbeale/code/samba/bin/default/third_party/uid_wrapper/libuid-wrapper.so --use-dns-faking --socket-wrapper fileserver && touch ./st/st_done) | /usr/bin/python3 -u /home/timbeale/code/samba/selftest/filter-subunit --expected-failures=/home/timbeale/code/samba/selftest/knownfail --expected-failures=/home/timbeale/code/samba/selftest/knownfail.d --flapping=/home/timbeale/code/samba/selftest/flapping --flapping=/home/timbeale/code/samba/selftest/flapping.d | tee ./st/subunit | /usr/bin/python3 -u /home/timbeale/code/samba/selftest/format-subunit --prefix=./st --immediate
Traceback (most recent call last):
Traceback (most recent call last):
File "/home/timbeale/code/samba/selftest/format-subunit", line 14, in <module>
File "/home/timbeale/code/samba/selftest/filter-subunit", line 27, in <module>
import subunithelper
import subunithelper
File "/home/timbeale/code/samba/selftest/subunithelper.py", line 25, in <module>
File "/home/timbeale/code/samba/selftest/subunithelper.py", line 25, in <module>
from samba import subunit
from samba import subunit
ImportError: cannot import name 'subunit'
ImportError: cannot import name 'subunit'
Traceback (most recent call last):
File "/home/timbeale/code/samba/selftest/tests.py", line 22, in <module>
from selftesthelpers import bindir, srcdir, python
File "/home/timbeale/code/samba/selftest/selftesthelpers.py", line 172, in <module>
smbtorture4_testsuite_list = subprocess.Popen([smbtorture4, "--list-suites"], stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate("")[0].decode('utf8').splitlines()
File "/usr/lib/python3.5/subprocess.py", line 947, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: '/home/timbeale/code/samba/bin/smbtorture'
Error creating recipe from /usr/bin/python3 /home/timbeale/code/samba/selftest/tests.py| at /home/timbeale/code/samba/selftest/selftest.pl line 694.
TOP 10 slowest tests
ERROR: test failed with exit code 1
Makefile:16: recipe for target 'test' failed
make: *** [test] Error 1
OK, let's try using PYTHONPATH to fix that:
PYTHONPATH=./python/ make test TESTS=fileserver
PYTHONHASHSEED=1 WAF_MAKE=1 ./buildtools/bin/waf test
'test' finished successfully (0.008s)
Waf: Entering directory `/home/timbeale/code/samba/bin/default'
Selected embedded Heimdal build
Waf: Leaving directory `/home/timbeale/code/samba/bin/default'
'build' finished successfully (3.182s)
test: running (/usr/bin/perl /home/timbeale/code/samba/selftest/selftest.pl --target=samba3 --prefix=./st --srcdir=/home/timbeale/code/samba --exclude=/home/timbeale/code/samba/selftest/skip --testlist="/usr/bin/python3 /home/timbeale/code/samba/selftest/tests.py|" --testlist="/usr/bin/python3 /home/timbeale/code/samba/source3/selftest/tests.py|" --testlist="/usr/bin/python3 /home/timbeale/code/samba/source4/selftest/tests.py|" --exclude=/home/timbeale/code/samba/selftest/slow --nss_wrapper_so_path=/home/timbeale/code/samba/bin/default/third_party/nss_wrapper/libnss-wrapper.so --resolv_wrapper_so_path=/home/timbeale/code/samba/bin/default/third_party/resolv_wrapper/libresolv-wrapper.so --socket_wrapper_so_path=/home/timbeale/code/samba/bin/default/third_party/socket_wrapper/libsocket-wrapper.so --uid_wrapper_so_path=/home/timbeale/code/samba/bin/default/third_party/uid_wrapper/libuid-wrapper.so --use-dns-faking --socket-wrapper fileserver && touch ./st/st_done) | /usr/bin/python3 -u /home/timbeale/code/samba/selftest/filter-subunit --expected-failures=/home/timbeale/code/samba/selftest/knownfail --expected-failures=/home/timbeale/code/samba/selftest/knownfail.d --flapping=/home/timbeale/code/samba/selftest/flapping --flapping=/home/timbeale/code/samba/selftest/flapping.d | tee ./st/subunit | /usr/bin/python3 -u /home/timbeale/code/samba/selftest/format-subunit --prefix=./st --immediate
Traceback (most recent call last):
File "/home/timbeale/code/samba/selftest/filter-subunit", line 27, in <module>
import subunithelper
File "/home/timbeale/code/samba/selftest/subunithelper.py", line 25, in <module>
from samba import subunit
File "/home/timbeale/code/samba/python/samba/__init__.py", line 28, in <module>
import ldb
ImportError: No module named 'ldb'
Traceback (most recent call last):
File "/home/timbeale/code/samba/selftest/format-subunit", line 14, in <module>
import subunithelper
File "/home/timbeale/code/samba/selftest/subunithelper.py", line 25, in <module>
from samba import subunit
File "/home/timbeale/code/samba/python/samba/__init__.py", line 28, in <module>
import ldb
ImportError: No module named 'ldb'
Traceback (most recent call last):
File "/home/timbeale/code/samba/selftest/tests.py", line 22, in <module>
from selftesthelpers import bindir, srcdir, python
File "/home/timbeale/code/samba/selftest/selftesthelpers.py", line 172, in <module>
smbtorture4_testsuite_list = subprocess.Popen([smbtorture4, "--list-suites"], stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate("")[0].decode('utf8').splitlines()
File "/usr/lib/python3.5/subprocess.py", line 947, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: '/home/timbeale/code/samba/bin/smbtorture'
Error creating recipe from /usr/bin/python3 /home/timbeale/code/samba/selftest/tests.py| at /home/timbeale/code/samba/selftest/selftest.pl line 694.
TOP 10 slowest tests
ERROR: test failed with exit code 1
Makefile:16: recipe for target 'test' failed
make: *** [test] Error 1
So trying to import subunit still tries to import the parent samba module, which tries to import ldb, which we haven't built.
There may be a simpler way to avoid this problem, but we can actually just move subunit into selftest, as it's fairly independent.
make test TESTS=fileserver
PYTHONHASHSEED=1 WAF_MAKE=1 ./buildtools/bin/waf test
'test' finished successfully (0.008s)
Waf: Entering directory `/home/timbeale/code/samba/bin/default'
Selected embedded Heimdal build
Waf: Leaving directory `/home/timbeale/code/samba/bin/default'
'build' finished successfully (3.188s)
test: running (/usr/bin/perl /home/timbeale/code/samba/selftest/selftest.pl --target=samba3 --prefix=./st --srcdir=/home/timbeale/code/samba --exclude=/home/timbeale/code/samba/selftest/skip --testlist="/usr/bin/python3 /home/timbeale/code/samba/selftest/tests.py|" --testlist="/usr/bin/python3 /home/timbeale/code/samba/source3/selftest/tests.py|" --testlist="/usr/bin/python3 /home/timbeale/code/samba/source4/selftest/tests.py|" --exclude=/home/timbeale/code/samba/selftest/slow --nss_wrapper_so_path=/home/timbeale/code/samba/bin/default/third_party/nss_wrapper/libnss-wrapper.so --resolv_wrapper_so_path=/home/timbeale/code/samba/bin/default/third_party/resolv_wrapper/libresolv-wrapper.so --socket_wrapper_so_path=/home/timbeale/code/samba/bin/default/third_party/socket_wrapper/libsocket-wrapper.so --uid_wrapper_so_path=/home/timbeale/code/samba/bin/default/third_party/uid_wrapper/libuid-wrapper.so --use-dns-faking --socket-wrapper fileserver && touch ./st/st_done) | /usr/bin/python3 -u /home/timbeale/code/samba/selftest/filter-subunit --expected-failures=/home/timbeale/code/samba/selftest/knownfail --expected-failures=/home/timbeale/code/samba/selftest/knownfail.d --flapping=/home/timbeale/code/samba/selftest/flapping --flapping=/home/timbeale/code/samba/selftest/flapping.d | tee ./st/subunit | /usr/bin/python3 -u /home/timbeale/code/samba/selftest/format-subunit --prefix=./st --immediate
Traceback (most recent call last):
File "/home/timbeale/code/samba/selftest/format-subunit", line 14, in <module>
import subunithelper
File "/home/timbeale/code/samba/selftest/subunithelper.py", line 25, in <module>
import subunit
File "/home/timbeale/code/samba/selftest/subunit/__init__.py", line 22, in <module>
samba.ensure_third_party_module("iso8601", "pyiso8601")
AttributeError: module 'samba' has no attribute 'ensure_third_party_module'
Traceback (most recent call last):
File "/home/timbeale/code/samba/selftest/filter-subunit", line 27, in <module>
import subunithelper
File "/home/timbeale/code/samba/selftest/subunithelper.py", line 25, in <module>
import subunit
File "/home/timbeale/code/samba/selftest/subunit/__init__.py", line 22, in <module>
samba.ensure_third_party_module("iso8601", "pyiso8601")
AttributeError: module 'samba' has no attribute 'ensure_third_party_module'
Traceback (most recent call last):
File "/home/timbeale/code/samba/selftest/tests.py", line 22, in <module>
from selftesthelpers import bindir, srcdir, python
File "/home/timbeale/code/samba/selftest/selftesthelpers.py", line 172, in <module>
smbtorture4_testsuite_list = subprocess.Popen([smbtorture4, "--list-suites"], stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate("")[0].decode('utf8').splitlines()
File "/usr/lib/python3.5/subprocess.py", line 947, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: '/home/timbeale/code/samba/bin/smbtorture'
Error creating recipe from /usr/bin/python3 /home/timbeale/code/samba/selftest/tests.py| at /home/timbeale/code/samba/selftest/selftest.pl line 694.
TOP 10 slowest tests
ERROR: test failed with exit code 1
Makefile:16: recipe for target 'test' failed
make: *** [test] Error 1
OK, so now it's trying to import the thirdparty iso8601 (which we haven't built), and it's using a whole pile of dependent samba python code to do so. For argument's sake, let's pretend we've solved these dependencies and just fix the import using PYTHONPATH...
PYTHONPATH=./third_party/pyiso8601/ make test TESTS=fileserver
PYTHONHASHSEED=1 WAF_MAKE=1 ./buildtools/bin/waf test
'test' finished successfully (0.008s)
Waf: Entering directory `/home/timbeale/code/samba/bin/default'
Selected embedded Heimdal build
Waf: Leaving directory `/home/timbeale/code/samba/bin/default'
'build' finished successfully (3.202s)
test: running (/usr/bin/perl /home/timbeale/code/samba/selftest/selftest.pl --target=samba3 --prefix=./st --srcdir=/home/timbeale/code/samba --exclude=/home/timbeale/code/samba/selftest/skip --testlist="/usr/bin/python3 /home/timbeale/code/samba/selftest/tests.py|" --testlist="/usr/bin/python3 /home/timbeale/code/samba/source3/selftest/tests.py|" --testlist="/usr/bin/python3 /home/timbeale/code/samba/source4/selftest/tests.py|" --exclude=/home/timbeale/code/samba/selftest/slow --nss_wrapper_so_path=/home/timbeale/code/samba/bin/default/third_party/nss_wrapper/libnss-wrapper.so --resolv_wrapper_so_path=/home/timbeale/code/samba/bin/default/third_party/resolv_wrapper/libresolv-wrapper.so --socket_wrapper_so_path=/home/timbeale/code/samba/bin/default/third_party/socket_wrapper/libsocket-wrapper.so --uid_wrapper_so_path=/home/timbeale/code/samba/bin/default/third_party/uid_wrapper/libuid-wrapper.so --use-dns-faking --socket-wrapper fileserver && touch ./st/st_done) | /usr/bin/python3 -u /home/timbeale/code/samba/selftest/filter-subunit --expected-failures=/home/timbeale/code/samba/selftest/knownfail --expected-failures=/home/timbeale/code/samba/selftest/knownfail.d --flapping=/home/timbeale/code/samba/selftest/flapping --flapping=/home/timbeale/code/samba/selftest/flapping.d | tee ./st/subunit | /usr/bin/python3 -u /home/timbeale/code/samba/selftest/format-subunit --prefix=./st --immediate
Traceback (most recent call last):
File "/home/timbeale/code/samba/selftest/format-subunit", line 14, in <module>
import subunithelper
File "/home/timbeale/code/samba/selftest/subunithelper.py", line 29, in <module>
from samba.compat import binary_type
ImportError: No module named 'samba.compat'
Traceback (most recent call last):
File "/home/timbeale/code/samba/selftest/filter-subunit", line 27, in <module>
import subunithelper
File "/home/timbeale/code/samba/selftest/subunithelper.py", line 29, in <module>
from samba.compat import binary_type
ImportError: No module named 'samba.compat'
Traceback (most recent call last):
File "/home/timbeale/code/samba/selftest/tests.py", line 22, in <module>
from selftesthelpers import bindir, srcdir, python
File "/home/timbeale/code/samba/selftest/selftesthelpers.py", line 172, in <module>
smbtorture4_testsuite_list = subprocess.Popen([smbtorture4, "--list-suites"], stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate("")[0].decode('utf8').splitlines()
File "/usr/lib/python3.5/subprocess.py", line 947, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: '/home/timbeale/code/samba/bin/smbtorture'
Error creating recipe from /usr/bin/python3 /home/timbeale/code/samba/selftest/tests.py| at /home/timbeale/code/samba/selftest/selftest.pl line 694.
TOP 10 slowest tests
ERROR: test failed with exit code 1
Makefile:16: recipe for target 'test' failed
make: *** [test] Error 1
OK, more samba python dependencies. However, the main problem we're hitting now is there's no bin/smbtorture. I believe it's not built because bld.PYTHON_BUILD_IS_ENABLED() is false in this case. It looks like smbtorture was one of the first things removed from the disable-python build - see commit 7000fa01f8844.
About this point I gave up.
More information about the samba-technical
mailing list