[SCM] Samba Shared Repository - branch master updated
Douglas Bagnall
dbagnall at samba.org
Fri Jun 23 04:46:04 UTC 2017
The branch, master has been updated
via 7f1ff6b samba_kcc: debugging: say intrasite when we mean intrasite
via ab40b40 samba_kcc: drop all connections from non-existent DSAs
via 6d78cbe samba_kcc: comment typo
via 414b7e9 third_party/dnspython: fix variable name in dnssec
via 822c4b9 samba_kcc: avoid crash on odd networks with --dot-file-dir
via 84ea6bc waf/wafadmin/3rdparty: fix paranoid.py variable names
via d5b123e python/getopt: -d/--debuglevel saves value in options for scripts
via 6cbd3b5 add provision performance tests
via c1e3937 perftest: add a new medley test
from 03042b8 Bug 15852. There are valid paths where conn->lsa_pipe_tcp->transport is NULL. Protect against this.
https://git.samba.org/?p=samba.git;a=shortlog;h=master
- Log -----------------------------------------------------------------
commit 7f1ff6bc8463ac56cced5e747b6ded8572a908fc
Author: Garming Sam <garming at catalyst.net.nz>
Date: Fri Jun 16 13:05:37 2017 +1200
samba_kcc: debugging: say intrasite when we mean intrasite
Signed-off-by: Garming Sam <garming at catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Autobuild-User(master): Douglas Bagnall <dbagnall at samba.org>
Autobuild-Date(master): Fri Jun 23 06:45:47 CEST 2017 on sn-devel-144
commit ab40b4013aef079e3a01b53435ee745fc21cc483
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Fri Jun 16 09:16:16 2017 +1200
samba_kcc: drop all connections from non-existent DSAs
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Garming Sam <garming at catalyst.net.nz>
commit 6d78cbe2faa8cc8ff38f84a7f6b65cc08cad25f5
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Fri Jun 16 09:15:17 2017 +1200
samba_kcc: comment typo
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Garming Sam <garming at catalyst.net.nz>
commit 414b7e939945e46003b36615a71d7195ccd15192
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Thu Mar 9 15:08:19 2017 +1300
third_party/dnspython: fix variable name in dnssec
This appears to have been fixed upstream (along with significant other
changes)
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Garming Sam <garming at catalyst.net.nz>
commit 822c4b9852ff59e984c29f98f2cd5200c00283b3
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Thu Apr 6 11:56:25 2017 +1200
samba_kcc: avoid crash on odd networks with --dot-file-dir
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Garming Sam <garming at catalyst.net.nz>
commit 84ea6bc645ac04fcd77941a4fe3f74b0e46e556f
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Thu Mar 9 14:56:24 2017 +1300
waf/wafadmin/3rdparty: fix paranoid.py variable names
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Garming Sam <garming at catalyst.net.nz>
commit d5b123e46161da9b3c1c9220bb79e985855bdf1e
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Wed Apr 12 12:34:49 2017 +1200
python/getopt: -d/--debuglevel saves value in options for scripts
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Garming Sam <garming at catalyst.net.nz>
commit 6cbd3b52cdac82b46ff31da3fc82211bf2a37ea9
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Wed Mar 8 15:25:53 2017 +1300
add provision performance tests
Because making provision faster makes autobuild faster.
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Garming Sam <garming at catalyst.net.nz>
commit c1e3937b3e47ae8e58b8dddba9ca67f3d7d709bf
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Wed Mar 22 10:25:06 2017 +1300
perftest: add a new medley test
This is something of a rewrite of ad_dc_performance.py with more
search tests and a rebalancing of others. For example, the users are
added in three lots of 2000 using varying methods rather than 5 of
1000 using ldap, reducing duplication thus clarifying the results.
Links are added in more realistic patterns with groups of varying
size.
To save time, the database is not cleaned up. Usually perftests are
run with TESTS= restriction to a single suite, but in case this is not
done, this suite is run last.
The ad_dc_performance suite is not replaced so that comparisons with
old test sequences are still possible.
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Garming Sam <garming at catalyst.net.nz>
-----------------------------------------------------------------------
Summary of changes:
python/samba/getopt.py | 1 +
python/samba/kcc/__init__.py | 9 +-
python/samba/kcc/graph_utils.py | 4 +
selftest/perf_tests.py | 20 +
.../dsdb/tests/python/ad_dc_medley_performance.py | 501 +++++++++++++++++++++
...ulti_bind.py => ad_dc_provision_performance.py} | 71 ++-
third_party/dnspython/dns/dnssec.py | 4 +-
third_party/waf/wafadmin/3rdparty/paranoid.py | 4 +-
8 files changed, 588 insertions(+), 26 deletions(-)
create mode 100644 source4/dsdb/tests/python/ad_dc_medley_performance.py
copy source4/dsdb/tests/python/{ad_dc_multi_bind.py => ad_dc_provision_performance.py} (50%)
Changeset truncated at 500 lines:
diff --git a/python/samba/getopt.py b/python/samba/getopt.py
index 4da75ce..9e1fb83 100644
--- a/python/samba/getopt.py
+++ b/python/samba/getopt.py
@@ -65,6 +65,7 @@ class SambaOptions(optparse.OptionGroup):
raise optparse.OptionValueError("invalid %s option value: %s" %
(opt_str, arg))
self._lp.set('debug level', str(arg))
+ parser.values.debuglevel = int(arg)
def _set_realm(self, option, opt_str, arg, parser):
self._lp.set('realm', arg)
diff --git a/python/samba/kcc/__init__.py b/python/samba/kcc/__init__.py
index f775a11..6f973ea 100644
--- a/python/samba/kcc/__init__.py
+++ b/python/samba/kcc/__init__.py
@@ -524,7 +524,7 @@ class KCC(object):
:return: None
"""
# TODO Figure out how best to handle the RODC case
- # The RODC is ITSG, but shouldn't act on anyone's behalf.
+ # The RODC is ISTG, but shouldn't act on anyone's behalf.
if self.my_dsa.is_ro():
return
@@ -541,8 +541,9 @@ class KCC(object):
if s_dnstr not in local_dsas:
from_dsa = self.get_dsa(s_dnstr)
# Samba ONLY: ISTG removes connections to dead DCs
- if from_dsa is None and '\\0ADEL' in s_dnstr:
- logger.info("DSA appears deleted, removing connection %s" % s_dnstr)
+ if from_dsa is None or '\\0ADEL' in s_dnstr:
+ logger.info("DSA appears deleted, removing connection %s"
+ % s_dnstr)
cn.to_be_deleted = True
continue
connections_and_dsas.append((cn, dsa, from_dsa))
@@ -2014,7 +2015,7 @@ class KCC(object):
if not needed:
debug.DEBUG_RED("%s lacks 'should be present' status, "
- "aborting construct_intersite_graph!" %
+ "aborting construct_intrasite_graph!" %
nc_x.nc_dnstr)
return
diff --git a/python/samba/kcc/graph_utils.py b/python/samba/kcc/graph_utils.py
index 704f70d..e326952 100644
--- a/python/samba/kcc/graph_utils.py
+++ b/python/samba/kcc/graph_utils.py
@@ -49,6 +49,10 @@ def write_dot_file(basename, edge_list, vertices=None, label=None,
for i, edge in enumerate(edge_list):
a, b = edge
+ if a is None:
+ a = "Missing source value"
+ if b is None:
+ b = "Missing destination value"
if reformat_labels:
a = a.replace(',', '\\n')
b = b.replace(',', '\\n')
diff --git a/selftest/perf_tests.py b/selftest/perf_tests.py
index adc4ef9..9b4289b 100644
--- a/selftest/perf_tests.py
+++ b/selftest/perf_tests.py
@@ -33,6 +33,14 @@ plantestsuite_loadlist("samba4.ndr_pack_performance.python(ad_dc_ntvfs)",
'--workgroup=$DOMAIN',
'$LOADLIST', '$LISTOPT'])
+plantestsuite_loadlist("samba4.provision_performance.python(ad_dc_ntvfs)",
+ "ad_dc_ntvfs",
+ [python, os.path.join(samba4srcdir,
+ "dsdb/tests/python/ad_dc_provision_performance.py"),
+ '$SERVER', '-U"$USERNAME%$PASSWORD"',
+ '--workgroup=$DOMAIN',
+ '$LOADLIST', '$LISTOPT'])
+
plantestsuite_loadlist("samba4.ldap.ad_dc_search_performance.python(ad_dc_ntvfs)",
"ad_dc_ntvfs",
[python,
@@ -64,3 +72,15 @@ plantestsuite_loadlist("samba4.ldb.multi_connect.python(ad_dc_ntvfs)",
"dsdb/tests/python/ad_dc_multi_bind.py"),
'tdb://$PREFIX_ABS/ad_dc_ntvfs/private/sam.ldb'
'$LOADLIST', '$LISTOPT'])
+
+
+# this one doesn't tidy itself up fully, so leave it as last unless
+# you want a messy database.
+plantestsuite_loadlist("samba4.ldap.ad_dc_medley_performance.python(ad_dc_ntvfs)",
+ "ad_dc_ntvfs",
+ [python,
+ os.path.join(samba4srcdir,
+ "dsdb/tests/python/ad_dc_medley_performance.py"),
+ '$SERVER', '-U"$USERNAME%$PASSWORD"',
+ '--workgroup=$DOMAIN',
+ '$LOADLIST', '$LISTOPT'])
diff --git a/source4/dsdb/tests/python/ad_dc_medley_performance.py b/source4/dsdb/tests/python/ad_dc_medley_performance.py
new file mode 100644
index 0000000..72655f7
--- /dev/null
+++ b/source4/dsdb/tests/python/ad_dc_medley_performance.py
@@ -0,0 +1,501 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+import optparse
+import sys
+sys.path.insert(0, 'bin/python')
+
+import os
+import samba
+import samba.getopt as options
+import random
+import tempfile
+import shutil
+import time
+import itertools
+
+from samba.netcmd.main import cmd_sambatool
+
+# We try to use the test infrastructure of Samba 4.3+, but if it
+# doesn't work, we are probably in a back-ported patch and trying to
+# run on 4.1 or something.
+#
+# Don't copy this horror into ordinary tests -- it is special for
+# performance tests that want to apply to old versions.
+try:
+ from samba.tests.subunitrun import SubunitOptions, TestProgram
+ ANCIENT_SAMBA = False
+except ImportError:
+ ANCIENT_SAMBA = True
+ samba.ensure_external_module("testtools", "testtools")
+ samba.ensure_external_module("subunit", "subunit/python")
+ from subunit.run import SubunitTestRunner
+ import unittest
+
+from samba.samdb import SamDB
+from samba.auth import system_session
+from ldb import Message, MessageElement, Dn, LdbError
+from ldb import FLAG_MOD_ADD, FLAG_MOD_REPLACE, FLAG_MOD_DELETE
+from ldb import SCOPE_BASE, SCOPE_SUBTREE, SCOPE_ONELEVEL
+
+parser = optparse.OptionParser("ad_dc_performance.py [options] <host>")
+sambaopts = options.SambaOptions(parser)
+parser.add_option_group(sambaopts)
+parser.add_option_group(options.VersionOptions(parser))
+
+if not ANCIENT_SAMBA:
+ subunitopts = SubunitOptions(parser)
+ parser.add_option_group(subunitopts)
+
+# use command line creds if available
+credopts = options.CredentialsOptions(parser)
+parser.add_option_group(credopts)
+opts, args = parser.parse_args()
+
+
+if len(args) < 1:
+ parser.print_usage()
+ sys.exit(1)
+
+host = args[0]
+
+lp = sambaopts.get_loadparm()
+creds = credopts.get_credentials(lp)
+
+random.seed(1)
+
+
+class PerfTestException(Exception):
+ pass
+
+
+BATCH_SIZE = 2000
+LINK_BATCH_SIZE = 1000
+DELETE_BATCH_SIZE = 50
+N_GROUPS = 29
+
+
+class GlobalState(object):
+ next_user_id = 0
+ n_groups = 0
+ next_linked_user = 0
+ next_relinked_user = 0
+ next_linked_user_3 = 0
+ next_removed_link_0 = 0
+ test_number = 0
+ active_links = set()
+
+class UserTests(samba.tests.TestCase):
+
+ def add_if_possible(self, *args, **kwargs):
+ """In these tests sometimes things are left in the database
+ deliberately, so we don't worry if we fail to add them a second
+ time."""
+ try:
+ self.ldb.add(*args, **kwargs)
+ except LdbError:
+ pass
+
+ def setUp(self):
+ super(UserTests, self).setUp()
+ self.state = GlobalState # the class itself, not an instance
+ self.lp = lp
+ self.ldb = SamDB(host, credentials=creds,
+ session_info=system_session(lp), lp=lp)
+ self.base_dn = self.ldb.domain_dn()
+ self.ou = "OU=pid%s,%s" % (os.getpid(), self.base_dn)
+ self.ou_users = "OU=users,%s" % self.ou
+ self.ou_groups = "OU=groups,%s" % self.ou
+ self.ou_computers = "OU=computers,%s" % self.ou
+
+ self.state.test_number += 1
+ random.seed(self.state.test_number)
+
+ def tearDown(self):
+ super(UserTests, self).tearDown()
+
+ def test_00_00_do_nothing(self):
+ # this gives us an idea of the overhead
+ pass
+
+ def test_00_01_do_nothing_relevant(self):
+ # takes around 1 second on i7-4770
+ j = 0
+ for i in range(30000000):
+ j += i
+
+ def test_00_02_do_nothing_sleepily(self):
+ time.sleep(1)
+
+ def test_00_03_add_ous_and_groups(self):
+ # initialise the database
+ for dn in (self.ou,
+ self.ou_users,
+ self.ou_groups,
+ self.ou_computers):
+ self.ldb.add({
+ "dn": dn,
+ "objectclass": "organizationalUnit"
+ })
+
+ for i in range(N_GROUPS):
+ self.ldb.add({
+ "dn": "cn=g%d,%s" % (i, self.ou_groups),
+ "objectclass": "group"
+ })
+
+ self.state.n_groups = N_GROUPS
+
+ def _add_users(self, start, end):
+ for i in range(start, end):
+ self.ldb.add({
+ "dn": "cn=u%d,%s" % (i, self.ou_users),
+ "objectclass": "user"
+ })
+
+ def _add_users_ldif(self, start, end):
+ lines = []
+ for i in range(start, end):
+ lines.append("dn: cn=u%d,%s" % (i, self.ou_users))
+ lines.append("objectclass: user")
+ lines.append("")
+ self.ldb.add_ldif('\n'.join(lines))
+
+ def _test_join(self):
+ tmpdir = tempfile.mkdtemp()
+ if '://' in host:
+ server = host.split('://', 1)[1]
+ else:
+ server = host
+ cmd = cmd_sambatool.subcommands['domain'].subcommands['join']
+ result = cmd._run("samba-tool domain join",
+ creds.get_realm(),
+ "dc", "-U%s%%%s" % (creds.get_username(),
+ creds.get_password()),
+ '--targetdir=%s' % tmpdir,
+ '--server=%s' % server)
+
+ shutil.rmtree(tmpdir)
+
+ def _test_unindexed_search(self):
+ expressions = [
+ ('(&(objectclass=user)(description='
+ 'Built-in account for adminstering the computer/domain))'),
+ '(description=Built-in account for adminstering the computer/domain)',
+ '(objectCategory=*)',
+ '(samaccountname=Administrator*)'
+ ]
+ for expression in expressions:
+ t = time.time()
+ for i in range(25):
+ self.ldb.search(self.ou,
+ expression=expression,
+ scope=SCOPE_SUBTREE,
+ attrs=['cn'])
+ print >> sys.stderr, '%d %s took %s' % (i, expression,
+ time.time() - t)
+
+ def _test_indexed_search(self):
+ expressions = ['(objectclass=group)',
+ '(samaccountname=Administrator)'
+ ]
+ for expression in expressions:
+ t = time.time()
+ for i in range(4000):
+ self.ldb.search(self.ou,
+ expression=expression,
+ scope=SCOPE_SUBTREE,
+ attrs=['cn'])
+ print >> sys.stderr, '%d runs %s took %s' % (i, expression,
+ time.time() - t)
+
+ def _test_base_search(self):
+ for dn in [self.base_dn, self.ou, self.ou_users,
+ self.ou_groups, self.ou_computers]:
+ for i in range(4000):
+ try:
+ self.ldb.search(dn,
+ scope=SCOPE_BASE,
+ attrs=['cn'])
+ except LdbError as (num, msg):
+ if num != 32:
+ raise
+
+ def _test_base_search_failing(self):
+ pattern = 'missing%d' + self.ou
+ for i in range(4000):
+ self.ldb.search(pattern % i,
+ scope=SCOPE_BASE,
+ attrs=['cn'])
+
+ def search_expression_list(self, expressions, rounds,
+ attrs=['cn'],
+ scope=SCOPE_SUBTREE):
+ for expression in expressions:
+ t = time.time()
+ for i in range(rounds):
+ self.ldb.search(self.ou,
+ expression=expression,
+ scope=SCOPE_SUBTREE,
+ attrs=['cn'])
+ print >> sys.stderr, '%d runs %s took %s' % (i, expression,
+ time.time() - t)
+
+ def _test_complex_search(self, n=100):
+ classes = ['samaccountname', 'objectCategory', 'dn', 'member']
+ values = ['*', '*t*', 'g*', 'user']
+ comparators = ['=', '<=', '>='] # '~=' causes error
+ maybe_not = ['!(', '']
+ joiners = ['&', '|']
+
+ # The number of permuations is 18432, which is not huge but
+ # would take hours to search. So we take a sample.
+ all_permutations = list(itertools.product(joiners,
+ classes, classes,
+ values, values,
+ comparators, comparators,
+ maybe_not, maybe_not))
+
+ expressions = []
+
+ for (j, c1, c2, v1, v2,
+ o1, o2, n1, n2) in random.sample(all_permutations, n):
+ expression = ''.join(['(', j,
+ '(', n1, c1, o1, v1,
+ '))' if n1 else ')',
+ '(', n2, c2, o2, v2,
+ '))' if n2 else ')',
+ ')'])
+ expressions.append(expression)
+
+ self.search_expression_list(expressions, 1)
+
+ def _test_member_search(self, rounds=10):
+ expressions = []
+ for d in range(20):
+ expressions.append('(member=cn=u%d,%s)' % (d + 500, self.ou_users))
+ expressions.append('(member=u%d*)' % (d + 700,))
+
+ self.search_expression_list(expressions, rounds)
+
+ def _test_memberof_search(self, rounds=200):
+ expressions = []
+ for i in range(min(self.state.n_groups, rounds)):
+ expressions.append('(memberOf=cn=g%d,%s)' % (i, self.ou_groups))
+ expressions.append('(memberOf=cn=g%d*)' % (i,))
+ expressions.append('(memberOf=cn=*%s*)' % self.ou_groups)
+
+ self.search_expression_list(expressions, 2)
+
+ def _test_add_many_users(self, n=BATCH_SIZE):
+ s = self.state.next_user_id
+ e = s + n
+ self._add_users(s, e)
+ self.state.next_user_id = e
+
+ def _test_add_many_users_ldif(self, n=BATCH_SIZE):
+ s = self.state.next_user_id
+ e = s + n
+ self._add_users_ldif(s, e)
+ self.state.next_user_id = e
+
+ def _link_user_and_group(self, u, g):
+ link = (u, g)
+ if link in self.state.active_links:
+ return False
+
+ m = Message()
+ m.dn = Dn(self.ldb, "CN=g%d,%s" % (g, self.ou_groups))
+ m["member"] = MessageElement("cn=u%d,%s" % (u, self.ou_users),
+ FLAG_MOD_ADD, "member")
+ self.ldb.modify(m)
+ self.state.active_links.add(link)
+ return True
+
+ def _unlink_user_and_group(self, u, g):
+ link = (u, g)
+ if link not in self.state.active_links:
+ return False
+
+ user = "cn=u%d,%s" % (u, self.ou_users)
+ group = "CN=g%d,%s" % (g, self.ou_groups)
+ m = Message()
+ m.dn = Dn(self.ldb, group)
+ m["member"] = MessageElement(user, FLAG_MOD_DELETE, "member")
+ self.ldb.modify(m)
+ self.state.active_links.remove(link)
+ return True
+
+ def _test_link_many_users(self, n=LINK_BATCH_SIZE):
+ # this links unevenly, putting more users in the first group
+ # and fewer in the last.
+ ng = self.state.n_groups
+ nu = self.state.next_user_id
+ while n:
+ u = random.randrange(nu)
+ g = random.randrange(random.randrange(ng) + 1)
+ if self._link_user_and_group(u, g):
+ n -= 1
+
+ def _test_link_many_users_batch(self, n=(LINK_BATCH_SIZE * 10)):
+ # this links unevenly, putting more users in the first group
+ # and fewer in the last.
+ ng = self.state.n_groups
+ nu = self.state.next_user_id
+ messages = []
+ for g in range(ng):
+ m = Message()
+ m.dn = Dn(self.ldb, "CN=g%d,%s" % (g, self.ou_groups))
+ messages.append(m)
+
+ while n:
+ u = random.randrange(nu)
+ g = random.randrange(random.randrange(ng) + 1)
+ link = (u, g)
+ if link in self.state.active_links:
+ continue
+ m = messages[g]
+ m["member%s" % u] = MessageElement("cn=u%d,%s" %
+ (u, self.ou_users),
+ FLAG_MOD_ADD, "member")
+ self.state.active_links.add(link)
+ n -= 1
+
+ for m in messages:
+ try:
+ self.ldb.modify(m)
+ except LdbError as e:
+ print e
+ print m
+
+ def _test_remove_some_links(self, n=(LINK_BATCH_SIZE // 2)):
+ victims = random.sample(list(self.state.active_links), n)
+ for x in victims:
+ self._unlink_user_and_group(*x)
+
+ test_00_11_join_empty_dc = _test_join
+
+ test_00_12_adding_users_2000 = _test_add_many_users
+
+ test_00_20_join_unlinked_2k_users = _test_join
+ test_00_21_unindexed_search_2k_users = _test_unindexed_search
+ test_00_22_indexed_search_2k_users = _test_indexed_search
+
+ test_00_23_complex_search_2k_users = _test_complex_search
+ test_00_24_member_search_2k_users = _test_member_search
+ test_00_25_memberof_search_2k_users = _test_memberof_search
+
+ test_00_27_base_search_2k_users = _test_base_search
+ test_00_28_base_search_failing_2k_users = _test_base_search_failing
+
+ test_01_01_link_2k_users = _test_link_many_users
+ test_01_02_link_2k_users_batch = _test_link_many_users_batch
+
+ test_02_10_join_2k_linked_dc = _test_join
+ test_02_11_unindexed_search_2k_linked_dc = _test_unindexed_search
+ test_02_12_indexed_search_2k_linked_dc = _test_indexed_search
+
+ test_04_01_remove_some_links_2k = _test_remove_some_links
+
--
Samba Shared Repository
More information about the samba-cvs
mailing list