[SCM] Samba Shared Repository - branch master updated
Stefan Metzmacher
metze at samba.org
Sun Jul 2 20:00:03 UTC 2017
The branch, master has been updated
via 0cfef7f selftest: Prime the netlogon cache during test_idmap_rfc2307
via eb26522 selftest: Bind rfc2307 tests to exactly one server
via 8b97a0a selftest: Use tree_delete control in idmap_rfc2307 test
via c0a7d2b selftest: Do not force run of kcc at start of selftest
via a6bd91f WHATSNEW: Add entry for Multi-process LDAP Server
via 9063669 WHATSNEW: Add an entry for the LDB whole DB locking issue
via f4ce778 ldap: Run the LDAP server with the default (typically standard) process model
via 5ec7246 dsdb: Improve debugging on start transacton failure
via d21d832 dsdb: Teach the Samba partition module how to lock all the DB backends
via 4894f47 dsdb: Add tests showing that the CN=CONFIGURATION partition is also locked
via b3db655 dsdb: Add new test adding a record to the top level sam.ldb file
via 4b5ff4a dsdb: Add more locking more tests, confirming blocking locks in both directions
via c5b4cbf dsdb: Add test showing a search can't start while a transaction is already repared in a backend partition
via c0e2909 dsdb: Add test showing a search can't start while a transaction is already repared
via 3f0a250 ldb: version 1.2.0
via 0fac7e4 ldb:includes: protect ldb_modules.h from being used by Samba < 4.7
via a1158d2 ldb:wscript: define EXPECTED_SYSTEM_LDB_VERSION_{MAJOR,MINOR,RELEASE}
via 774b2e4 ldb:wscript: provide LDB_VERSION_{MAJOR,MINOR,RELEASE} in ldb_version.h
via b8b8439 ldb:tests: Extend api.py testsuite to show transaction_commit() blocks against the whole-db read lock
via 6d7208c ldb:tests: Extend api.py testsuite to show transaction contents can not be seen outside the transaction
via 86e8425 ldb:tests: Add test to show that locks are released on TALLOC_FREE(req)
via 4b7c274 ldb:tests: Correct comment about version numbers
via a895851 ldb: Lock the whole backend database for the duration of a search
via d3c0df3 ldb_tdb: Implement read_lock and read_unlock module operations
via 6a3e77e ldb: Add read_lock and read_unlock to ldb_module_ops
via 31ef1d6 ldb:tests: Add test encoding current locking behaviour during ldb_search()
via 0cd75c1 ldb:tests: Show that writes do not appear during an ldb_search()
via fd0beef ldb:tests: don't assert the results before doing the final search finished
via 289d7d8 ldb:tdb: Ensure we correctly decrement ltdb->read_lock_count
via 25e580f ldb_pack: use ldb_dn_from_ldb_val() and avoid a duplicate strlen() call
via 49c45fbb4 tdb: version 1.3.14
via 32702a9 tdb: Add new function tdb_transaction_active()
via e08d02d tdb: Improve documentation for tdb_transaction_start()
via f80076f tdb: Remove locking from tdb_traverse_read()
via b87af32 drsuapi: Improve debugging in DsAddEntry()
via 4ca48ee wafsamba: add maxversion and version_blacklist to CHECK_BUNDLED_SYSTEM[_PKG]()
from 51c1a2a s3: VFS: Change SMB_VFS_CONNECTPATH to take const struct smb_filename * instead of const char *.
https://git.samba.org/?p=samba.git;a=shortlog;h=master
- Log -----------------------------------------------------------------
commit 0cfef7f50efcb5e1c9d1c35e54c29796b6aabe26
Author: Andrew Bartlett <abartlet at samba.org>
Date: Sat Jul 1 22:20:17 2017 +1200
selftest: Prime the netlogon cache during test_idmap_rfc2307
This ensures that the group memberships just created are reflected in the test
comparison. Otherwise we are trusting that no caches are primed, which is
simply not safe in a test.
(The login will put a list of groups, as obtained by the login over NETLOGON or
via the PAC, into the samlogon cache).
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
Autobuild-User(master): Stefan Metzmacher <metze at samba.org>
Autobuild-Date(master): Sun Jul 2 21:59:18 CEST 2017 on sn-devel-144
commit eb26522a5265ea9be4e31beb5cd819fb15316f28
Author: Andrew Bartlett <abartlet at samba.org>
Date: Sat Jul 1 21:44:38 2017 +1200
selftest: Bind rfc2307 tests to exactly one server
The tests make changes to the DC, do not wait for replication, then expect
those to be reflected in the client. If they bind to another server this
will not hold true.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
commit 8b97a0af326edc275eb6ede9b2edc7720dc808ff
Author: Andrew Bartlett <abartlet at samba.org>
Date: Sat Jul 1 21:34:44 2017 +1200
selftest: Use tree_delete control in idmap_rfc2307 test
This control removes an entire subtree, which was the intention of the previouse code
but much more effectively.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
commit c0a7d2bacdacc7a43e04f3b450ea069c000b3d8d
Author: Bob Campbell <bobcampbell at catalyst.net.nz>
Date: Tue Jan 10 09:20:47 2017 +1300
selftest: Do not force run of kcc at start of selftest
This should help to avoid clashes between periodic and manual runs of
the KCC during autobuild.
Pair-Programmed-With: Andrew Bartlett <abartlet at samba.org>
Signed-off-by: Bob Campbell <bobcampbell at catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
commit a6bd91f4ec3c329d6363710cc50662460697f630
Author: Andrew Bartlett <abartlet at samba.org>
Date: Fri Jun 30 16:02:46 2017 +1200
WHATSNEW: Add entry for Multi-process LDAP Server
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
commit 9063669a05a261657d5b9a60254bd1b9065e6423
Author: Andrew Bartlett <abartlet at samba.org>
Date: Thu Jun 29 12:50:03 2017 +1200
WHATSNEW: Add an entry for the LDB whole DB locking issue
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
commit f4ce77857bb677ea612ad26d700960f913ff7bd8
Author: Andrew Bartlett <abartlet at samba.org>
Date: Mon Oct 17 13:55:42 2016 +1300
ldap: Run the LDAP server with the default (typically standard) process model
This allows one LDAP socket to proceed if another fails, and reduces the
impact of a crash becoming a DoS bug, as it only impacts one socket.
This may mean we have a lot of idle tasks, but this should not be a big
issue
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
commit 5ec724687e1a714b2effbfb47e997632e3346297
Author: Andrew Bartlett <abartlet at samba.org>
Date: Wed Jun 28 17:34:05 2017 +1200
dsdb: Improve debugging on start transacton failure
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
commit d21d83219534782c5d75a707aab23a5d7e91ce56
Author: Andrew Bartlett <abartlet at samba.org>
Date: Mon Jun 26 14:13:41 2017 +1200
dsdb: Teach the Samba partition module how to lock all the DB backends
The metadata partition (sam.ldb) lock is not
enough to block another process in prepare_commit(),
because prepare_commit() is a no-op, if nothing
was changed in the specific backend.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
commit 4894f47e2ec48dd36cac72608bfabde6fdfc7c6b
Author: Andrew Bartlett <abartlet at samba.org>
Date: Mon Jun 26 13:34:21 2017 +1200
dsdb: Add tests showing that the CN=CONFIGURATION partition is also locked
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
commit b3db6558d324c23ad8011175d9cb0b70ad233121
Author: Andrew Bartlett <abartlet at samba.org>
Date: Mon Jun 26 13:16:01 2017 +1200
dsdb: Add new test adding a record to the top level sam.ldb file
This shows that locks are made on this file as well
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
commit 4b5ff4a3092aa7700e0193a00882261ee990aa10
Author: Stefan Metzmacher <metze at samba.org>
Date: Fri Jun 23 12:13:19 2017 +0200
dsdb: Add more locking more tests, confirming blocking locks in both directions
These extended tests allow us to show that a search (read) blocks a
transaction commit (write), and that a transaction commit blocks a
search.
Pair-Programmed-With: Andrew Bartlett <abartlet at samba.org>
Signed-off-by: Stefan Metzmacher <metze at samba.org>
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
commit c5b4cbf34e21ac77124a2ca421c1fdd179329783
Author: Andrew Bartlett <abartlet at samba.org>
Date: Fri Jun 16 15:49:45 2017 +1200
dsdb: Add test showing a search can't start while a transaction is already repared in a backend partition
Pair-programmed-with: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
commit c0e2909595b22fd489571f60b364878b7c484fd3
Author: Andrew Bartlett <abartlet at samba.org>
Date: Fri Jun 16 15:49:45 2017 +1200
dsdb: Add test showing a search can't start while a transaction is already repared
Pair-programmed-with: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
commit 3f0a2500cec6aaf0104c3b61a8c93d09db80365f
Author: Stefan Metzmacher <metze at samba.org>
Date: Tue Apr 11 17:50:08 2017 +0200
ldb: version 1.2.0
* handle one more LDB_FLAG_INTERNAL_DISABLE_SINGLE_VALUE_CHECK
case in ldb_tdb
* fix ldb_tdb locking (performance) problems
* fix ldb_tdb search inconsistencies by adding
read_[un]lock() hooks to the module stack
(bug #12858)
* add cmocka based tests for the locking issues
* ldb_version.h provides LDB_VERSION_{MAJOR,MINOR,RELEASE} defines
* protect ldb_modules.h from being used by Samba < 4.7
Note: that this release (as well as 1.1.30 and 1.1.31)
may cause problems for older applications, e.g. Samba
See https://bugzilla.samba.org/show_bug.cgi?id=12859
Signed-off-by: Stefan Metzmacher <metze at samba.org>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 0fac7e4d7391f3f4aed54f0c7fbc9fd3cc33bf6a
Author: Stefan Metzmacher <metze at samba.org>
Date: Fri Jun 30 08:14:02 2017 +0200
ldb:includes: protect ldb_modules.h from being used by Samba < 4.7
Samba versions before 4.7 are incompatible with the read_[un]lock()
behaviour introduced into ldb.
This makes sure older Samba versions fail to compile against
ldb >= 1.2.0.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12859
Signed-off-by: Stefan Metzmacher <metze at samba.org>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit a1158d223711b82eb8d8312fb3f0709d307c0fb7
Author: Stefan Metzmacher <metze at samba.org>
Date: Fri Jun 30 08:09:38 2017 +0200
ldb:wscript: define EXPECTED_SYSTEM_LDB_VERSION_{MAJOR,MINOR,RELEASE}
This indicates what feature set Samba assumes from the used
libldb from the system.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12859
Signed-off-by: Stefan Metzmacher <metze at samba.org>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 774b2e4f5a77a7996f453b5d7d76514a80011c51
Author: Stefan Metzmacher <metze at samba.org>
Date: Fri Jun 23 10:50:54 2017 +0200
ldb:wscript: provide LDB_VERSION_{MAJOR,MINOR,RELEASE} in ldb_version.h
Signed-off-by: Stefan Metzmacher <metze at samba.org>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit b8b8439f1a7213d03e251ddc0279ae8a2be414d3
Author: Andrew Bartlett <abartlet at samba.org>
Date: Fri Jun 16 15:49:16 2017 +1200
ldb:tests: Extend api.py testsuite to show transaction_commit() blocks against the whole-db read lock
The new ldb whole-db lock behaviour now allows this test
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
commit 6d7208c36fcf104eb07405e1d02e3e167efd854d
Author: Andrew Bartlett <abartlet at samba.org>
Date: Fri Jun 16 15:44:46 2017 +1200
ldb:tests: Extend api.py testsuite to show transaction contents can not be seen outside the transaction
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
commit 86e84254180d7fb8ed1762682b2c32578bfa86c7
Author: Andrew Bartlett <abartlet at samba.org>
Date: Fri Jun 16 12:19:00 2017 +1200
ldb:tests: Add test to show that locks are released on TALLOC_FREE(req)
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
commit 4b7c274cc0e8110dbd4342c3255972496ea5ced8
Author: Andrew Bartlett <abartlet at samba.org>
Date: Fri Jun 16 12:18:39 2017 +1200
ldb:tests: Correct comment about version numbers
(ldb releases have been made while this patch set was in train)
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
commit a8958515f0340d42e64837fcdf5823f04f6f2544
Author: Andrew Bartlett <abartlet at samba.org>
Date: Thu Jun 15 13:56:46 2017 +1200
ldb: Lock the whole backend database for the duration of a search
We must hold locks not just for the duration of each search, but for the whole search
as our module stack may make multiple search requests to build up the whole result.
This is explains a number of replication and read corruption issues in Samba
Pair-Programmed-With: Stefan Metzmacher <metze at samba.org>
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Signed-off-by: Stefan Metzmacher <metze at samba.org>
commit d3c0df35c9af535148b91f87de75aaf4c2e57b71
Author: Andrew Bartlett <abartlet at samba.org>
Date: Fri May 12 01:39:08 2017 +0200
ldb_tdb: Implement read_lock and read_unlock module operations
This allows Samba to provide a consistent view of the DB
despite the use of multiple databases via the partitions module
and over multiple callbacks via a module stack.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
commit 6a3e77edc31b356865c5d7b9e6e1567b2d48c9f6
Author: Andrew Bartlett <abartlet at samba.org>
Date: Thu Jun 15 12:10:51 2017 +1200
ldb: Add read_lock and read_unlock to ldb_module_ops
This will be used to implement read locking in ldb_tdb
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
commit 31ef1d6211f4bc448beca8da5d584624ba1ed68f
Author: Andrew Bartlett <abartlet at samba.org>
Date: Mon May 22 16:18:20 2017 +1200
ldb:tests: Add test encoding current locking behaviour during ldb_search()
Currently, a lock is not held against modifications once the final
record is returned via a callback, so modifications can be made
during the DONE callback. This makes it hard to write modules
that interpert an ldb search result and do further processing
so will change in the future to allow the full search to be
atomic.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
commit 0cd75c1fb1fc29cde581f1546e12de823f82af47
Author: Andrew Bartlett <abartlet at samba.org>
Date: Tue Apr 25 22:33:53 2017 +1200
ldb:tests: Show that writes do not appear during an ldb_search()
A modify or rename during a search must not cause a search to change
output, and attributes having an index should in particular not see
any change in behaviour in this respect
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
commit fd0beef179001dac213366998a5ba3b6e3ed787b
Author: Andrew Bartlett <abartlet at samba.org>
Date: Thu Jun 15 13:56:46 2017 +1200
ldb:tests: don't assert the results before doing the final search finished
This is required to pass the test in future, because
otherwise the clean up will fail because we hold locks.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
commit 289d7d8fc5c2fcc004eeb0f0e4ce475c33e3952e
Author: Garming Sam <garming at catalyst.net.nz>
Date: Thu Mar 30 12:03:17 2017 +1300
ldb:tdb: Ensure we correctly decrement ltdb->read_lock_count
If we do not do this, then we never take the all record lock, and instead do a lock
for every record as we go, which is very slow during a large search
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Signed-off-by: Garming Sam <garming at catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
commit 25e580f86e82c41a9380e42ad16d94053abaa8d4
Author: Andrew Bartlett <abartlet at samba.org>
Date: Fri Jun 30 12:55:15 2017 +1200
ldb_pack: use ldb_dn_from_ldb_val() and avoid a duplicate strlen() call
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
commit 49c45fbb41f30c6a4d532039c811b43c7284e0f6
Author: Stefan Metzmacher <metze at samba.org>
Date: Tue Apr 11 17:27:33 2017 +0200
tdb: version 1.3.14
* allow tdb_traverse_read before tdb_transaction[_prepare]_commit()
* Improve documentation for tdb_transaction_start()
* Add new function tdb_transaction_active()
Signed-off-by: Stefan Metzmacher <metze at samba.org>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 32702a974579ae1ee2d097240dc2bf55c515ca57
Author: Andrew Bartlett <abartlet at samba.org>
Date: Thu Apr 27 08:34:56 2017 +1200
tdb: Add new function tdb_transaction_active()
This will allow callers to avoid their own reference counting of transactions.
Additionally, this will always line up with the acutal transaction state, even
in the error cases where tdb can cancel the transaction
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
commit e08d02d0279f3a69735b7139d9219b603c68b149
Author: Andrew Bartlett <abartlet at samba.org>
Date: Thu Apr 27 08:51:08 2017 +1200
tdb: Improve documentation for tdb_transaction_start()
It now references the TDB_ALLOW_NESTING and TDB_DISALLOW_NESTING flags
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
commit f80076fe43f18404575cb9ac86ecfc637127c4ba
Author: Andrew Bartlett <abartlet at samba.org>
Date: Fri Mar 31 17:34:13 2017 +1300
tdb: Remove locking from tdb_traverse_read()
This restores the original intent of tdb_traverse_read() in
7dd31288a701d772e45b1960ac4ce4cc1be782ed
This is needed to avoid a deadlock with tdb_lockall() and the
transaction start, as ldb_tdb should take the allrecord lock during a
search (which calls tdb_traverse), and can otherwise deadlock against
a transaction starting in another process
We add a test to show that a transaction can now start while a read
traverse is in progress
This allows more operations to happen in parallel. The blocking point
is moved to the prepare commit.
This in turn permits a roughly doubling of unindexed search
performance, because currently ldb_tdb omits to take the lock due to
an unrelated bug, but taking the allrecord lock triggers the
above-mentioned deadlock.
This behaviour was added in 251aaafe3a9213118ac3a92def9ab2104c40d12a for
Solaris 10 in 2005. But the run-fcntl-deadlock test works also on Solaris 10,
see https://lists.samba.org/archive/samba-technical/2017-April/119876.html.
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
commit b87af322d5fea8bd8280057996ce542c999bcaef
Author: Andrew Bartlett <abartlet at samba.org>
Date: Wed May 3 06:34:01 2017 +0200
drsuapi: Improve debugging in DsAddEntry()
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Stefan Metzmacher <metze at samba.org>
commit 4ca48ee4d060f773dcdf9f78a5e4c1b1263b61f4
Author: Stefan Metzmacher <metze at samba.org>
Date: Fri Jun 30 06:21:32 2017 +0200
wafsamba: add maxversion and version_blacklist to CHECK_BUNDLED_SYSTEM[_PKG]()
BUG: https://bugzilla.samba.org/show_bug.cgi?id=12859
Signed-off-by: Stefan Metzmacher <metze at samba.org>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
-----------------------------------------------------------------------
Summary of changes:
WHATSNEW.txt | 41 +
buildtools/wafsamba/samba_bundled.py | 21 +-
lib/ldb/ABI/{ldb-1.1.31.sigs => ldb-1.2.0.sigs} | 2 +
...yldb-util-1.1.10.sigs => pyldb-util-1.2.0.sigs} | 0
...-util-1.1.10.sigs => pyldb-util.py3-1.2.0.sigs} | 0
lib/ldb/common/ldb.c | 159 +++-
lib/ldb/common/ldb_modules.c | 46 ++
lib/ldb/common/ldb_pack.c | 5 +-
lib/ldb/include/ldb_module.h | 39 +
lib/ldb/ldb_tdb/ldb_tdb.c | 3 +
lib/ldb/tests/ldb_mod_op_test.c | 906 +++++++++++++++++++--
lib/ldb/tests/python/api.py | 84 +-
lib/ldb/wscript | 26 +-
lib/tdb/ABI/{tdb-1.3.12.sigs => tdb-1.3.14.sigs} | 1 +
lib/tdb/common/transaction.c | 8 +
lib/tdb/common/traverse.c | 10 +-
lib/tdb/include/tdb.h | 29 +-
lib/tdb/test/run-nested-traverse.c | 31 +-
lib/tdb/test/run-traverse-in-transaction.c | 3 +
lib/tdb/wscript | 2 +-
nsswitch/tests/test_idmap_rfc2307.sh | 20 +-
python/samba/tests/dsdb.py | 358 +++++++-
selftest/target/Samba3.pm | 2 +
selftest/target/Samba4.pm | 53 +-
source4/dsdb/repl/replicated_objects.c | 3 +-
source4/dsdb/samdb/ldb_modules/partition.c | 176 +++-
source4/ldap_server/ldap_server.c | 9 +-
source4/rpc_server/drsuapi/addentry.c | 2 +
28 files changed, 1888 insertions(+), 151 deletions(-)
copy lib/ldb/ABI/{ldb-1.1.31.sigs => ldb-1.2.0.sigs} (99%)
copy lib/ldb/ABI/{pyldb-util-1.1.10.sigs => pyldb-util-1.2.0.sigs} (100%)
copy lib/ldb/ABI/{pyldb-util-1.1.10.sigs => pyldb-util.py3-1.2.0.sigs} (100%)
copy lib/tdb/ABI/{tdb-1.3.12.sigs => tdb-1.3.14.sigs} (98%)
Changeset truncated at 500 lines:
diff --git a/WHATSNEW.txt b/WHATSNEW.txt
index e8fbecb..dbca75e 100644
--- a/WHATSNEW.txt
+++ b/WHATSNEW.txt
@@ -45,6 +45,37 @@ so it's still possible to connect to SMB1-only servers by default.
NEW FEATURES/CHANGES
====================
+Whole DB read locks: Improved LDAP and replication consistency
+--------------------------------------------------------------
+
+Prior to Samba 4.7 and ldb 1.2.0, the LDB database layer used by Samba
+erronously did not take whole-DB read locks to protect search
+and DRS replication operations.
+
+While each object returned remained subject to a record-level lock (so
+would remain consistent to itself), under a race condition with a
+rename or delete, it and any links (like the member attribute) to it
+would not be returned.
+
+The symptoms of this issue include:
+
+Replication failures with this error showing in the client side logs:
+ error during DRS repl ADD: No objectClass found in replPropertyMetaData for
+ Failed to commit objects:
+ WERR_GEN_FAILURE/NT_STATUS_INVALID_NETWORK_RESPONSE
+
+A crash of the server, in particular the rpc_server process with
+ INTERNAL ERROR: Signal 11
+
+LDAP read inconsistency
+ A DN subject to a search at the same time as it is being renamed
+ may not appear under either the old or new name, but will re-appear
+ for a subsequent search.
+
+See https://bugzilla.samba.org/show_bug.cgi?id=12858 for more details
+and updated advise on database recovery for affected installations.
+
+
Samba AD with MIT Kerberos
--------------------------
@@ -105,6 +136,16 @@ authentication, SMB and RPC authorization is covered, however password
changes are not at this stage, and this support is not currently
backed by a testsuite.
+Multi-process LDAP Server
+-------------------------
+
+The LDAP server in the AD DC now honours the process model used for
+the rest of the samba process, rather than being forced into a single
+process. This aids in Samba's ability to scale to larger numbers of AD
+clients and the AD DC's overall resiliency, but will mean that there is a
+fork()ed child for every LDAP client, which may be more resource
+intensive in some situations.
+
Query record for open file or directory
---------------------------------------
diff --git a/buildtools/wafsamba/samba_bundled.py b/buildtools/wafsamba/samba_bundled.py
index ea88807..aa6199e 100644
--- a/buildtools/wafsamba/samba_bundled.py
+++ b/buildtools/wafsamba/samba_bundled.py
@@ -110,6 +110,7 @@ def LIB_MUST_BE_PRIVATE(conf, libname):
@conf
def CHECK_BUNDLED_SYSTEM_PKG(conf, libname, minversion='0.0.0',
+ maxversion=None, version_blacklist=[],
onlyif=None, implied_deps=None, pkg=None):
'''check if a library is available as a system library.
@@ -117,12 +118,15 @@ def CHECK_BUNDLED_SYSTEM_PKG(conf, libname, minversion='0.0.0',
'''
return conf.CHECK_BUNDLED_SYSTEM(libname,
minversion=minversion,
+ maxversion=maxversion,
+ version_blacklist=version_blacklist,
onlyif=onlyif,
implied_deps=implied_deps,
pkg=pkg)
@conf
def CHECK_BUNDLED_SYSTEM(conf, libname, minversion='0.0.0',
+ maxversion=None, version_blacklist=[],
checkfunctions=None, headers=None, checkcode=None,
onlyif=None, implied_deps=None,
require_headers=True, pkg=None, set_target=True):
@@ -181,16 +185,29 @@ def CHECK_BUNDLED_SYSTEM(conf, libname, minversion='0.0.0',
minversion = minimum_library_version(conf, libname, minversion)
msg = 'Checking for system %s' % libname
+ msg_ver = []
if minversion != '0.0.0':
- msg += ' >= %s' % minversion
+ msg_ver.append('>=%s' % minversion)
+ if maxversion is not None:
+ msg_ver.append('<=%s' % maxversion)
+ for v in version_blacklist:
+ msg_ver.append('!=%s' % v)
+ if msg_ver != []:
+ msg += " (%s)" % (" ".join(msg_ver))
uselib_store=libname.upper()
if pkg is None:
pkg = libname
+ version_checks = '%s >= %s' % (pkg, minversion)
+ if maxversion is not None:
+ version_checks += ' %s <= %s' % (pkg, maxversion)
+ for v in version_blacklist:
+ version_checks += ' %s != %s' % (pkg, v)
+
# try pkgconfig first
if (conf.CHECK_CFG(package=pkg,
- args='"%s >= %s" --cflags --libs' % (pkg, minversion),
+ args='"%s" --cflags --libs' % (version_checks),
msg=msg, uselib_store=uselib_store) and
check_functions_headers_code()):
if set_target:
diff --git a/lib/ldb/ABI/ldb-1.1.31.sigs b/lib/ldb/ABI/ldb-1.2.0.sigs
similarity index 99%
copy from lib/ldb/ABI/ldb-1.1.31.sigs
copy to lib/ldb/ABI/ldb-1.2.0.sigs
index d183708..1be2ae7 100644
--- a/lib/ldb/ABI/ldb-1.1.31.sigs
+++ b/lib/ldb/ABI/ldb-1.2.0.sigs
@@ -186,6 +186,8 @@ ldb_next_del_trans: int (struct ldb_module *)
ldb_next_end_trans: int (struct ldb_module *)
ldb_next_init: int (struct ldb_module *)
ldb_next_prepare_commit: int (struct ldb_module *)
+ldb_next_read_lock: int (struct ldb_module *)
+ldb_next_read_unlock: int (struct ldb_module *)
ldb_next_remote_request: int (struct ldb_module *, struct ldb_request *)
ldb_next_request: int (struct ldb_module *, struct ldb_request *)
ldb_next_start_trans: int (struct ldb_module *)
diff --git a/lib/ldb/ABI/pyldb-util-1.1.10.sigs b/lib/ldb/ABI/pyldb-util-1.2.0.sigs
similarity index 100%
copy from lib/ldb/ABI/pyldb-util-1.1.10.sigs
copy to lib/ldb/ABI/pyldb-util-1.2.0.sigs
diff --git a/lib/ldb/ABI/pyldb-util-1.1.10.sigs b/lib/ldb/ABI/pyldb-util.py3-1.2.0.sigs
similarity index 100%
copy from lib/ldb/ABI/pyldb-util-1.1.10.sigs
copy to lib/ldb/ABI/pyldb-util.py3-1.2.0.sigs
diff --git a/lib/ldb/common/ldb.c b/lib/ldb/common/ldb.c
index 700d89c..a4d9977 100644
--- a/lib/ldb/common/ldb.c
+++ b/lib/ldb/common/ldb.c
@@ -966,6 +966,146 @@ static int ldb_msg_check_element_flags(struct ldb_context *ldb,
return LDB_SUCCESS;
}
+/*
+ * This context allows us to make the unlock be a talloc destructor
+ *
+ * This ensures that a request started, but not waited on, will still
+ * unlock.
+ */
+struct ldb_db_lock_context {
+ struct ldb_request *req;
+ struct ldb_context *ldb;
+};
+
+/*
+ * We have to have a the unlock on a destructor so that we unlock the
+ * DB if a caller calls talloc_free(req). We trust that the ldb
+ * context has not already gone away.
+ */
+static int ldb_db_lock_destructor(struct ldb_db_lock_context *lock_context)
+{
+ int ret;
+ struct ldb_module *next_module;
+ FIRST_OP_NOERR(lock_context->ldb, read_unlock);
+ if (next_module != NULL) {
+ ret = next_module->ops->read_unlock(next_module);
+ } else {
+ ret = LDB_SUCCESS;
+ }
+
+ if (ret != LDB_SUCCESS) {
+ ldb_debug(lock_context->ldb,
+ LDB_DEBUG_FATAL,
+ "Failed to unlock db: %s / %s",
+ ldb_errstring(lock_context->ldb),
+ ldb_strerror(ret));
+ }
+ return 0;
+}
+
+static int ldb_lock_backend_callback(struct ldb_request *req,
+ struct ldb_reply *ares)
+{
+ struct ldb_db_lock_context *lock_context;
+ int ret;
+
+ lock_context = talloc_get_type(req->context,
+ struct ldb_db_lock_context);
+
+ if (!ares) {
+ return ldb_module_done(lock_context->req, NULL, NULL,
+ LDB_ERR_OPERATIONS_ERROR);
+ }
+ if (ares->error != LDB_SUCCESS || ares->type == LDB_REPLY_DONE) {
+ ret = ldb_module_done(lock_context->req, ares->controls,
+ ares->response, ares->error);
+ /*
+ * If this is a LDB_REPLY_DONE or an error, unlock the
+ * DB by calling the destructor on this context
+ */
+ talloc_free(lock_context);
+ return ret;
+ }
+
+ /* Otherwise pass on the callback */
+ switch (ares->type) {
+ case LDB_REPLY_ENTRY:
+ return ldb_module_send_entry(lock_context->req, ares->message,
+ ares->controls);
+
+ case LDB_REPLY_REFERRAL:
+ return ldb_module_send_referral(lock_context->req,
+ ares->referral);
+ default:
+ /* Can't happen */
+ return LDB_ERR_OPERATIONS_ERROR;
+ }
+}
+
+/*
+ * Do an ldb_search() with a lock held, but release it if the request
+ * is freed with talloc_free()
+ */
+static int lock_search(struct ldb_module *lock_module, struct ldb_request *req)
+{
+ /* Used in FIRST_OP_NOERR to find where to send the lock request */
+ struct ldb_module *next_module = NULL;
+ struct ldb_request *down_req = NULL;
+ struct ldb_db_lock_context *lock_context;
+ struct ldb_context *ldb = ldb_module_get_ctx(lock_module);
+ int ret;
+
+ lock_context = talloc(req, struct ldb_db_lock_context);
+ if (lock_context == NULL) {
+ return ldb_oom(ldb);
+ }
+
+ lock_context->ldb = ldb;
+ lock_context->req = req;
+
+ ret = ldb_build_search_req_ex(&down_req, ldb, req,
+ req->op.search.base,
+ req->op.search.scope,
+ req->op.search.tree,
+ req->op.search.attrs,
+ req->controls,
+ lock_context,
+ ldb_lock_backend_callback,
+ req);
+ LDB_REQ_SET_LOCATION(down_req);
+ if (ret != LDB_SUCCESS) {
+ return ret;
+ }
+
+ /* call DB lock */
+ FIRST_OP_NOERR(ldb, read_lock);
+ if (next_module != NULL) {
+ ret = next_module->ops->read_lock(next_module);
+ } else {
+ ret = LDB_ERR_UNSUPPORTED_CRITICAL_EXTENSION;
+ }
+
+ if (ret == LDB_ERR_UNSUPPORTED_CRITICAL_EXTENSION) {
+ /* We might be talking LDAP */
+ ldb_reset_err_string(ldb);
+ ret = 0;
+ TALLOC_FREE(lock_context);
+
+ return ldb_next_request(lock_module, req);
+ } else if ((ret != LDB_SUCCESS) && (ldb->err_string == NULL)) {
+ /* if no error string was setup by the backend */
+ ldb_asprintf_errstring(ldb, "Failed to get DB lock: %s (%d)",
+ ldb_strerror(ret), ret);
+ } else {
+ talloc_set_destructor(lock_context, ldb_db_lock_destructor);
+ }
+
+ if (ret != LDB_SUCCESS) {
+ return ret;
+ }
+
+ return ldb_next_request(lock_module, down_req);
+}
/*
start an ldb request
@@ -991,15 +1131,32 @@ int ldb_request(struct ldb_context *ldb, struct ldb_request *req)
/* call the first module in the chain */
switch (req->operation) {
case LDB_SEARCH:
+ {
+ /*
+ * A fake module to allow ldb_next_request() to be
+ * re-used and to keep the locking out of this function.
+ */
+ static const struct ldb_module_ops lock_module_ops = {
+ .name = "lock_searches",
+ .search = lock_search
+ };
+ struct ldb_module lock_module = {
+ .ldb = ldb,
+ .next = ldb->modules,
+ .ops = &lock_module_ops
+ };
+ next_module = &lock_module;
+
/* due to "ldb_build_search_req" base DN always != NULL */
if (!ldb_dn_validate(req->op.search.base)) {
ldb_asprintf_errstring(ldb, "ldb_search: invalid basedn '%s'",
ldb_dn_get_linearized(req->op.search.base));
return LDB_ERR_INVALID_DN_SYNTAX;
}
- FIRST_OP(ldb, search);
+
ret = next_module->ops->search(next_module, req);
break;
+ }
case LDB_ADD:
if (!ldb_dn_validate(req->op.add.message->dn)) {
ldb_asprintf_errstring(ldb, "ldb_add: invalid dn '%s'",
diff --git a/lib/ldb/common/ldb_modules.c b/lib/ldb/common/ldb_modules.c
index ca93299..3dd0438 100644
--- a/lib/ldb/common/ldb_modules.c
+++ b/lib/ldb/common/ldb_modules.c
@@ -641,6 +641,52 @@ int ldb_next_end_trans(struct ldb_module *module)
return ret;
}
+int ldb_next_read_lock(struct ldb_module *module)
+{
+ int ret;
+ FIND_OP(module, read_lock);
+ ret = module->ops->read_lock(module);
+ if (ret == LDB_SUCCESS) {
+ return ret;
+ }
+ if (!ldb_errstring(module->ldb)) {
+ /* Set a default error string, to place the blame somewhere */
+ ldb_asprintf_errstring(module->ldb,
+ "read_lock error in module %s: %s (%d)",
+ module->ops->name, ldb_strerror(ret),
+ ret);
+ }
+ if ((module && module->ldb->flags & LDB_FLG_ENABLE_TRACING)) {
+ ldb_debug(module->ldb, LDB_DEBUG_TRACE,
+ "ldb_next_read_lock error: %s",
+ ldb_errstring(module->ldb));
+ }
+ return ret;
+}
+
+int ldb_next_read_unlock(struct ldb_module *module)
+{
+ int ret;
+ FIND_OP(module, read_unlock);
+ ret = module->ops->read_unlock(module);
+ if (ret == LDB_SUCCESS) {
+ return ret;
+ }
+ if (!ldb_errstring(module->ldb)) {
+ /* Set a default error string, to place the blame somewhere */
+ ldb_asprintf_errstring(module->ldb,
+ "read_unlock error in module %s: %s (%d)",
+ module->ops->name, ldb_strerror(ret),
+ ret);
+ }
+ if ((module && module->ldb->flags & LDB_FLG_ENABLE_TRACING)) {
+ ldb_debug(module->ldb, LDB_DEBUG_TRACE,
+ "ldb_next_read_unlock error: %s",
+ ldb_errstring(module->ldb));
+ }
+ return ret;
+}
+
int ldb_next_prepare_commit(struct ldb_module *module)
{
int ret;
diff --git a/lib/ldb/common/ldb_pack.c b/lib/ldb/common/ldb_pack.c
index a63dd58..1f1688a 100644
--- a/lib/ldb/common/ldb_pack.c
+++ b/lib/ldb/common/ldb_pack.c
@@ -274,7 +274,10 @@ int ldb_unpack_data_only_attr_list_flags(struct ldb_context *ldb,
if (flags & LDB_UNPACK_DATA_FLAG_NO_DN) {
message->dn = NULL;
} else {
- message->dn = ldb_dn_new(message, ldb, (char *)p);
+ struct ldb_val blob;
+ blob.data = discard_const_p(uint8_t, p);
+ blob.length = dn_len;
+ message->dn = ldb_dn_from_ldb_val(message, ldb, &blob);
if (message->dn == NULL) {
errno = ENOMEM;
goto failed;
diff --git a/lib/ldb/include/ldb_module.h b/lib/ldb/include/ldb_module.h
index 3d56e68..8ad212a 100644
--- a/lib/ldb/include/ldb_module.h
+++ b/lib/ldb/include/ldb_module.h
@@ -35,6 +35,41 @@
#include <ldb.h>
+#if defined(_SAMBA_BUILD_) && defined(USING_SYSTEM_LDB)
+
+/*
+ * Versions before 1.2.0 doesn't define these values
+ * so we assime 1.1.29 (which was used in Samba 4.6)
+ *
+ * See https://bugzilla.samba.org/show_bug.cgi?id=12859
+ */
+#ifndef EXPECTED_SYSTEM_LDB_VERSION_MAJOR
+#define EXPECTED_SYSTEM_LDB_VERSION_MAJOR 1
+#endif
+#ifndef EXPECTED_SYSTEM_LDB_VERSION_MINOR
+#define EXPECTED_SYSTEM_LDB_VERSION_MINOR 1
+#endif
+#ifndef EXPECTED_SYSTEM_LDB_VERSION_MINOR
+#define EXPECTED_SYSTEM_LDB_VERSION_MINOR 29
+#endif
+
+/*
+ * Only Samba versions which expect ldb >= 1.2.0
+ * are compatible with read_[un]lock() behaviour.
+ *
+ * See https://bugzilla.samba.org/show_bug.cgi?id=12859
+ */
+#if EXPECTED_SYSTEM_LDB_VERSION_MAJOR > 1
+#define __LDB_READ_LOCK_COMPATIBLE__ 1
+#elif EXPECTED_SYSTEM_LDB_VERSION_MINOR > 1
+#define __LDB_READ_LOCK_COMPATIBLE__ 1
+#endif
+#ifndef __LDB_READ_LOCK_COMPATIBLE__
+#error "Samba < 4.7 is not compatible with this version of ldb due to assumptions around read locks"
+#endif
+
+#endif /* defined(_SAMBA_BUILD_) && defined(USING_SYSTEM_LDB) */
+
struct ldb_context;
struct ldb_module;
@@ -77,6 +112,8 @@ struct ldb_module_ops {
int (*end_transaction)(struct ldb_module *);
int (*del_transaction)(struct ldb_module *);
int (*sequence_number)(struct ldb_module *, struct ldb_request *);
+ int (*read_lock)(struct ldb_module *);
+ int (*read_unlock)(struct ldb_module *);
void *private_data;
};
@@ -203,6 +240,8 @@ int ldb_next_end_trans(struct ldb_module *module);
int ldb_next_del_trans(struct ldb_module *module);
int ldb_next_prepare_commit(struct ldb_module *module);
int ldb_next_init(struct ldb_module *module);
+int ldb_next_read_lock(struct ldb_module *module);
+int ldb_next_read_unlock(struct ldb_module *module);
void ldb_set_errstring(struct ldb_context *ldb, const char *err_string);
void ldb_asprintf_errstring(struct ldb_context *ldb, const char *format, ...) PRINTF_ATTRIBUTE(2,3);
diff --git a/lib/ldb/ldb_tdb/ldb_tdb.c b/lib/ldb/ldb_tdb/ldb_tdb.c
index f470e02..2ac1967 100644
--- a/lib/ldb/ldb_tdb/ldb_tdb.c
+++ b/lib/ldb/ldb_tdb/ldb_tdb.c
@@ -119,6 +119,7 @@ int ltdb_unlock_read(struct ldb_module *module)
struct ltdb_private *ltdb = talloc_get_type(data, struct ltdb_private);
if (ltdb->in_transaction == 0 && ltdb->read_lock_count == 1) {
tdb_unlockall_read(ltdb->tdb);
+ ltdb->read_lock_count--;
return 0;
}
ltdb->read_lock_count--;
@@ -1576,6 +1577,8 @@ static const struct ldb_module_ops ltdb_ops = {
.end_transaction = ltdb_end_trans,
.prepare_commit = ltdb_prepare_commit,
.del_transaction = ltdb_del_trans,
+ .read_lock = ltdb_lock_read,
+ .read_unlock = ltdb_unlock_read,
};
/*
diff --git a/lib/ldb/tests/ldb_mod_op_test.c b/lib/ldb/tests/ldb_mod_op_test.c
index fc4b134..96d2dd2 100644
--- a/lib/ldb/tests/ldb_mod_op_test.c
+++ b/lib/ldb/tests/ldb_mod_op_test.c
@@ -1285,7 +1285,7 @@ static void test_search_match_basedn(void **state)
* - (2) the ldb_transaction_commit() is called.
* This returns LDB_ERR_BUSY if the deadlock is detected
*
- * With ldb 1.1.29 and tdb 1.3.12 we avoid this only due to a missing
+ * With ldb 1.1.31 and tdb 1.3.12 we avoid this only due to a missing
* lock call in ltdb_search() due to a refcounting bug in
* ltdb_lock_read()
*/
@@ -1327,7 +1327,7 @@ static int test_ldb_search_against_transaction_callback2(struct ldb_request *req
--
Samba Shared Repository
More information about the samba-cvs
mailing list