[SCM] Samba Shared Repository - branch master updated
Ralph Böhme
slow at samba.org
Mon Feb 11 13:11:02 UTC 2019
The branch, master has been updated
via b1740f3bafc CI: split out "samba-ad-dc-ntvfs[-py2]" test targets
via 09c4e78fefd dsdb/tests/vlv: use only one toplevel dn that is correctly cleaned up
via 162219de1df blackbox/dbcheck.sh: fix dbcheck_fix_one_way_links cleanup
via 80d293ca0c4 selftest: improve debugging in dns_hub.py
via 55cce815876 selftest: Make dns_hub socket timeout match DNS_REQUEST_TIMEOUT
from 2e8daeb2bb3 docs-xml/smbdotconf: document export of SAMBA_CPS_{ACCOUNT,USER_PRINCIPAL,FULL}_NAME for check password script
https://git.samba.org/?p=samba.git;a=shortlog;h=master
- Log -----------------------------------------------------------------
commit b1740f3bafc2d460c137ac9f03514d57138d2f7a
Author: Ralph Boehme <slow at samba.org>
Date: Wed Jan 23 09:43:33 2019 +0100
CI: split out "samba-ad-dc-ntvfs[-py2]" test targets
Many AD tests currently use the "samba" target. Split out a new target
"samba-ad-dc-ntvfs" and have all tests that use the "ad_dc_ntvfs" env
use the new target. This should greatly speed up the runtime for the "samba"
target and avoid swapping.
This reduces the total CI time by ~ 55%, I got an autobuild and a gitlab
pipeline finished in just ~ 100 mins!
Pair-Programmed-With: Stefan Metzmacher <metze at samba.org>
Signed-off-by: Ralph Boehme <slow at samba.org>
Signed-off-by: Stefan Metzmacher <metze at samba.org>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
Autobuild-User(master): Ralph Böhme <slow at samba.org>
Autobuild-Date(master): Mon Feb 11 14:10:12 CET 2019 on sn-devel-144
commit 09c4e78fefd55d33f9fc2d3c3e1ba092a8cc490a
Author: Stefan Metzmacher <metze at samba.org>
Date: Fri Feb 8 10:57:13 2019 +0100
dsdb/tests/vlv: use only one toplevel dn that is correctly cleaned up
Before "OU=vlvtestou2,%s" % (self.base_dn) was left behind after the
test.
Signed-off-by: Stefan Metzmacher <metze at samba.org>
Reviewed-by: Ralph Boehme <slow at samba.org>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 162219de1df60573d0e42b73162fd5d086a812ad
Author: Stefan Metzmacher <metze at samba.org>
Date: Fri Feb 8 00:19:56 2019 +0100
blackbox/dbcheck.sh: fix dbcheck_fix_one_way_links cleanup
Commit 35bfc62a31c9ad73449594ddd48f76f50e0abade changed
dbcheck to not regard old one-way-links as errors.
At that time the relavant trigger changed from
fix_all_string_dn_component_mismatch to
fix_all_old_dn_string_component_mismatch.
Signed-off-by: Stefan Metzmacher <metze at samba.org>
Reviewed-by: Ralph Boehme <slow at samba.org>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 80d293ca0c40b63251e0dd1f059129a24b5c241f
Author: Stefan Metzmacher <metze at samba.org>
Date: Wed Jan 23 09:34:40 2019 +0100
selftest: improve debugging in dns_hub.py
We only print debug messages when the response is delayed by more than 2
seconds.
Signed-off-by: Stefan Metzmacher <metze at samba.org>
Reviewed-by: Ralph Boehme <slow at samba.org>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 55cce815876a235f9a5e6015a96d1984bcf3a4ed
Author: Tim Beale <timbeale at catalyst.net.nz>
Date: Mon Feb 4 09:28:07 2019 +1300
selftest: Make dns_hub socket timeout match DNS_REQUEST_TIMEOUT
I was hitting the recv_packet = s.recv(2048, 0) exception because
the socket timeout was reached. We've seen it before, but it seemed more
common after changing the default process-model to prefork. This patch
makes the socket timeout used by the python code consistent with the C
code.
Signed-off-by: Tim Beale <timbeale at catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
Reviewed-by: Ralph Boehme <slow at samba.org>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
-----------------------------------------------------------------------
Summary of changes:
.gitlab-ci.yml | 10 +++++++++
script/autobuild.py | 26 +++++++++++++++++-----
selftest/target/dns_hub.py | 47 +++++++++++++++++++++++++++++++++-------
source4/dsdb/tests/python/vlv.py | 18 +++++++--------
testprogs/blackbox/dbcheck.sh | 2 +-
5 files changed, 79 insertions(+), 24 deletions(-)
Changeset truncated at 500 lines:
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 908c29ec9d9..0729599e1b8 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -118,3 +118,13 @@ build_samba_buildpy2_only:
script:
- python script/autobuild.py samba-buildpy2-only --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
+build_samba_ad_dc_ntvfs:
+ <<: *shared_template
+ script:
+ # this one takes about 100 mins to finish
+ - script/autobuild.py samba-ad-dc-ntvfs --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
+
+build_samba_ad_dc_ntvfs_py2:
+ <<: *shared_template
+ script:
+ - script/autobuild.py samba-ad-dc-ntvfs-py2 --verbose --nocleanup --keeplogs --tail --testbase /tmp/samba-testbase
diff --git a/script/autobuild.py b/script/autobuild.py
index 00f0d2202a3..b1bd9d1686c 100755
--- a/script/autobuild.py
+++ b/script/autobuild.py
@@ -49,6 +49,8 @@ builddirs = {
"samba-none-env": ".",
"samba-ad-dc": ".",
"samba-ad-dc-py2": ".",
+ "samba-ad-dc-ntvfs": ".",
+ "samba-ad-dc-ntvfs-py2": ".",
"samba-ad-dc-2": ".",
"samba-ad-dc-2-py2": ".",
"samba-ad-dc-backup": ".",
@@ -93,8 +95,10 @@ tasks = {
("check-clean-tree", "../script/clean-source-tree.sh", "text/plain"),
("clean", "make clean", "text/plain")],
- # We have 'test' before 'install' because, 'test' should work without 'install (runs ad_dc_ntvfs and all the other envs)'
- "samba": [("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
+ # We have 'test' before 'install' because, 'test' should work without 'install (runs all the other envs)'
+ "samba": [
+ ("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
+ ("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
("make", "make -j", "text/plain"),
("test", "make test FAIL_IMMEDIATELY=1 "
"TESTS='${PY3_ONLY}"
@@ -102,6 +106,7 @@ tasks = {
"--exclude-env=nt4_dc "
"--exclude-env=nt4_member "
"--exclude-env=ad_dc "
+ "--exclude-env=ad_dc_ntvfs "
"--exclude-env=ad_dc_no_nss "
"--exclude-env=fl2003dc "
"--exclude-env=fl2008r2dc "
@@ -123,7 +128,6 @@ tasks = {
("check-clean-tree", "script/clean-source-tree.sh", "text/plain"),
("clean", "make clean", "text/plain")],
- # We split out this so the isolated nt4_dc tests do not wait for ad_dc or ad_dc_ntvfs tests (which are long)
"samba-nt4": [("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
("configure", "./configure.developer --without-ads --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
("make", "make -j", "text/plain"),
@@ -134,7 +138,6 @@ tasks = {
("check-clean-tree", "script/clean-source-tree.sh", "text/plain"),
("clean", "make clean", "text/plain")],
- # We split out this so the isolated ad_dc tests do not wait for ad_dc_ntvfs tests (which are long)
"samba-fileserver": [("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
("configure", "./configure.developer --without-ad-dc --without-ldap --without-ads --without-json --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
("make", "make -j", "text/plain"),
@@ -143,7 +146,6 @@ tasks = {
"--include-env=fileserver'", "text/plain"),
("check-clean-tree", "script/clean-source-tree.sh", "text/plain")],
- # We split out this so the isolated ad_dc tests do not wait for ad_dc_ntvfs tests (which are long)
"samba-ad-dc": [("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
("make", "make -j", "text/plain"),
@@ -157,7 +159,6 @@ tasks = {
"--include-env=ad_member_idmap_ad'", "text/plain"),
("check-clean-tree", "script/clean-source-tree.sh", "text/plain")],
- # We split out this so the isolated ad_dc tests do not wait for ad_dc_ntvfs tests (which are long)
"samba-ad-dc-2": [("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
("make", "make -j", "text/plain"),
@@ -171,6 +172,19 @@ tasks = {
"text/plain"),
("check-clean-tree", "script/clean-source-tree.sh", "text/plain")],
+ # We split out the ad_dc_ntvfs tests (which are long) so other test do not wait
+ # This is currently the longest task, so we don't randomly delay it.
+ "samba-ad-dc-ntvfs": [
+ ("random-sleep", "script/random-sleep.sh 1 1", "text/plain"),
+ ("configure", "./configure.developer --with-selftest-prefix=./bin/ab" + samba_configure_params, "text/plain"),
+ ("make", "make -j", "text/plain"),
+ ("test", "make test FAIL_IMMEDIATELY=1 "
+ "TESTS='${PY3_ONLY}"
+ "--include-env=ad_dc_ntvfs "
+ "'",
+ "text/plain"),
+ ("check-clean-tree", "script/clean-source-tree.sh", "text/plain")],
+
# run the backup/restore testenvs separately as they're fairly standalone
# (and CI seems to max out at ~8 different DCs running at once)
"samba-ad-dc-backup": [("random-sleep", "script/random-sleep.sh 60 600", "text/plain"),
diff --git a/selftest/target/dns_hub.py b/selftest/target/dns_hub.py
index d2d1f39e752..2cfc8d076aa 100755
--- a/selftest/target/dns_hub.py
+++ b/selftest/target/dns_hub.py
@@ -24,6 +24,7 @@ import threading
import sys
import select
import socket
+import time
from samba.dcerpc import dns
import samba.ndr as ndr
@@ -34,15 +35,27 @@ else:
import socketserver
sserver = socketserver
+DNS_REQUEST_TIMEOUT = 10
+
class DnsHandler(sserver.BaseRequestHandler):
+ dns_qtype_strings = dict((v, k) for k, v in vars(dns).items() if k.startswith('DNS_QTYPE_'))
+ def dns_qtype_string(self, qtype):
+ "Return a readable qtype code"
+ return self.dns_qtype_strings[qtype]
+
+ dns_rcode_strings = dict((v, k) for k, v in vars(dns).items() if k.startswith('DNS_RCODE_'))
+ def dns_rcode_string(self, rcode):
+ "Return a readable error code"
+ return self.dns_rcode_strings[rcode]
+
def dns_transaction_udp(self, packet, host):
"send a DNS query and read the reply"
s = None
try:
send_packet = ndr.ndr_pack(packet)
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, 0)
- s.settimeout(5)
+ s.settimeout(DNS_REQUEST_TIMEOUT)
s.connect((host, 53))
s.sendall(send_packet, 0)
recv_packet = s.recv(2048, 0)
@@ -63,6 +76,13 @@ class DnsHandler(sserver.BaseRequestHandler):
return 'ignore'
if lname.endswith('dsfsdfs'):
return 'fail'
+ if lname.endswith("torture1", 0, len(lname)-2):
+ # CATCH TORTURE100, TORTURE101, ...
+ return 'torture'
+ if lname.endswith('_none_.example.com'):
+ return 'torture'
+ if lname.endswith('torturedom.samba.example.com'):
+ return 'torture'
if lname.endswith('adnonssdom.samba.example.com'):
return '127.0.0.17'
if lname.endswith('adnontlmdom.samba.example.com'):
@@ -90,6 +110,7 @@ class DnsHandler(sserver.BaseRequestHandler):
return None
def handle(self):
+ start = time.monotonic()
data, sock = self.request
query = ndr.ndr_unpack(dns.name_packet, data)
name = query.questions[0].name
@@ -100,13 +121,13 @@ class DnsHandler(sserver.BaseRequestHandler):
return
elif forwarder is 'fail':
pass
- elif forwarder is not None:
- response = self.dns_transaction_udp(query, forwarder)
- else:
+ elif forwarder in ['torture', None]:
response = query
response.operation |= dns.DNS_FLAG_REPLY
response.operation |= dns.DNS_FLAG_RECURSION_AVAIL
response.operation |= dns.DNS_RCODE_NXDOMAIN
+ else:
+ response = self.dns_transaction_udp(query, forwarder)
if response is None:
response = query
@@ -116,14 +137,24 @@ class DnsHandler(sserver.BaseRequestHandler):
send_packet = ndr.ndr_pack(response)
- print("dns_hub: sending %s to address %s for name %s\n" %
- (forwarder, self.client_address, name))
+ end = time.monotonic()
+ tdiff = end - start
+ errcode = response.operation & dns.DNS_RCODE
+ if tdiff > (DNS_REQUEST_TIMEOUT/5):
+ debug = True
+ else:
+ debug = False
+ if debug:
+ print("dns_hub: forwarder[%s] client[%s] name[%s][%s] %s response.operation[0x%x] tdiff[%s]\n" %
+ (forwarder, self.client_address, name,
+ self.dns_qtype_string(query.questions[0].question_type),
+ self.dns_rcode_string(errcode), response.operation, tdiff))
try:
sock.sendto(send_packet, self.client_address)
except socket.error as err:
- print("Error sending %s to address %s for name %s: %s\n" %
- (forwarder, self.client_address, name, err))
+ print("dns_hub: Error sending response to client[%s] for name[%s] tdiff[%s]: %s\n" %
+ (self.client_address, name, tdiff, err))
class server_thread(threading.Thread):
diff --git a/source4/dsdb/tests/python/vlv.py b/source4/dsdb/tests/python/vlv.py
index 8550a38e287..29cc1718c78 100644
--- a/source4/dsdb/tests/python/vlv.py
+++ b/source4/dsdb/tests/python/vlv.py
@@ -153,12 +153,16 @@ class TestsWithUserOU(samba.tests.TestCase):
session_info=system_session(lp), lp=lp)
self.base_dn = self.ldb.domain_dn()
- self.ou = "ou=vlv,%s" % self.base_dn
+ self.tree_dn = "ou=vlvtesttree,%s" % self.base_dn
+ self.ou = "ou=vlvou,%s" % self.tree_dn
if opts.delete_in_setup:
try:
- self.ldb.delete(self.ou, ['tree_delete:1'])
+ self.ldb.delete(self.tree_dn, ['tree_delete:1'])
except ldb.LdbError as e:
- print("tried deleting %s, got error %s" % (self.ou, e))
+ print("tried deleting %s, got error %s" % (self.tree_dn, e))
+ self.ldb.add({
+ "dn": self.tree_dn,
+ "objectclass": "organizationalUnit"})
self.ldb.add({
"dn": self.ou,
"objectclass": "organizationalUnit"})
@@ -191,7 +195,7 @@ class TestsWithUserOU(samba.tests.TestCase):
def tearDown(self):
super(TestsWithUserOU, self).tearDown()
if not opts.delete_in_setup:
- self.ldb.delete(self.ou, ['tree_delete:1'])
+ self.ldb.delete(self.tree_dn, ['tree_delete:1'])
class VLVTests(TestsWithUserOU):
@@ -1368,11 +1372,7 @@ class PagedResultsTests(TestsWithUserOU):
def test_paged_modify_object_scope(self):
expr = "(objectClass=*)"
- ou2 = "OU=vlvtestou2,%s" % (self.base_dn)
- try:
- self.ldb.delete(ou2, ['tree_delete:1'])
- except ldb.LdbError:
- pass
+ ou2 = "OU=vlvtestou2,%s" % (self.tree_dn)
self.ldb.add({"dn": ou2, "objectclass": "organizationalUnit"})
# Do a separate, full search to get all results
diff --git a/testprogs/blackbox/dbcheck.sh b/testprogs/blackbox/dbcheck.sh
index 5691426d01b..caf89ca402b 100755
--- a/testprogs/blackbox/dbcheck.sh
+++ b/testprogs/blackbox/dbcheck.sh
@@ -19,7 +19,7 @@ dbcheck() {
# This list of attributes can be freely extended
dbcheck_fix_one_way_links() {
- $PYTHON $BINDIR/samba-tool dbcheck --quiet --fix --yes fix_all_string_dn_component_mismatch --attrs="lastKnownParent defaultObjectCategory fromServer rIDSetReferences" --cross-ncs $ARGS
+ $PYTHON $BINDIR/samba-tool dbcheck --quiet --fix --yes fix_all_old_dn_string_component_mismatch --attrs="lastKnownParent defaultObjectCategory fromServer rIDSetReferences" --cross-ncs $ARGS
}
# This list of attributes can be freely extended
--
Samba Shared Repository
More information about the samba-cvs
mailing list