[SCM] Samba Shared Repository - branch v4-9-test updated
Karolin Seeger
kseeger at samba.org
Thu Mar 28 13:47:02 UTC 2019
The branch, v4-9-test has been updated
via 5b7161153d0 s3:waf: Fix the detection of makdev() macro on Linux
via 055b971a7b0 regfio tests: Update comment style to match README.Coding
via 0cc3508242b regfio: Update code near recent changes to match README.Coding
via f3552ad511c regfio: Improve handling of malformed registry hive files
via b5ae06cc653 regfio: Add trivial unit test
via 223352ee944 regfio: Use correct function names in debug information
via 4644b23b91c Fix typos in "valid"
via 87ffad41af1 py/kcc_utils: py2.6 compatibility
via d44f2157a72 py/graph: use 2.6 compatible check for set membership
via 42b62465fcc dbcheck: use the str() value of the "name" attribute
via 693c349874f dbcheck: don't check expired tombstone objects by default anymore
via 3fca3dcc1c9 blackbox/dbcheck-links.sh: prepare regression test for skipping expired tombstones
via 543fc3e9c04 blackbox/dbcheck*.sh: pass --selftest-check-expired-tombstones to dbcheck
via 02f3d0a1a2c dbcheck: add --selftest-check-expired-tombstones cmdline option
via aebf46d957f python/samba/netcmd: provide SUPPRESS_HELP via Option class
via 107883dff6c dbcheck: detect the change after deletion bug
via 860b04aa7ae blackbox/dbcheck-links.sh: add regression test for lost deleted object repair
via 45850169a9c dbcheck: add find_repl_attid() helper function
via 7402d9cfcf7 dbcheck: don't remove dangling one-way links on already deleted objects
via 07ebd654a01 dbcheck: don't move already deleted objects to LostAndFound
via 76de43f052f dbcheck: do isDeleted, systemFlags and replPropertyMetaData detection first
via 0aaf7c98bb7 dbcheck: use DSDB_CONTROL_DBCHECK_FIX_LINK_DN_NAME when renaming deleted objects
via 44c83b09c60 dsdb:repl_meta_data: allow CONTROL_DBCHECK_FIX_LINK_DN_NAME to by pass rename
via 9339b096793 blackbox/dbcheck-links.sh: reproduce lost deleted object problem
via 7bcb0729652 selftest: force running with TZ=UTC
via 5602db1b1d5 python/samba: extra ndr_unpack needs bytes function
via 139da67cb3b python/samba: PY3 port for ridalloc_exop test to work
from b861e5e91f5 s4:librpc: Fix installation of Samba
https://git.samba.org/?p=samba.git;a=shortlog;h=v4-9-test
- Log -----------------------------------------------------------------
commit 5b7161153d0799adc249e99cb16b9b0cdbde896a
Author: Andreas Schneider <asn at samba.org>
Date: Thu Mar 21 11:55:46 2019 +0100
s3:waf: Fix the detection of makdev() macro on Linux
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13853
Signed-off-by: Andreas Schneider <asn at samba.org>
Reviewed-by: Jeremy Allison <jra at samba.org>
(cherry picked from commit eace58b539a382c61edd7c2be6fdfab31114719f)
Autobuild-User(v4-9-test): Karolin Seeger <kseeger at samba.org>
Autobuild-Date(v4-9-test): Thu Mar 28 13:46:27 UTC 2019 on sn-devel-144
commit 055b971a7b0e91f00d29873b58a3596a9313ee23
Author: Andrew Bartlett <abartlet at samba.org>
Date: Wed Mar 20 17:33:46 2019 +1300
regfio tests: Update comment style to match README.Coding
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13840
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
(cherry picked from commit 68c0fc4335d0c3c526a38481538a33290be6d58a)
commit 0cc3508242b82727e03147d2e1c414558d6b23b6
Author: Andrew Bartlett <abartlet at samba.org>
Date: Wed Mar 20 17:32:39 2019 +1300
regfio: Update code near recent changes to match README.Coding
This file long predates our current code conventions.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13840
Signed-off-by: Andrew Bartlett <abartlet at samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
(cherry picked from commit acbf103fcaa4150a57bfbab2450e36b5b39e399b)
commit f3552ad511c8c2a343dd503c0faf3ea8410cf895
Author: Michael Hanselmann <public at hansmi.ch>
Date: Sun Mar 17 13:49:20 2019 +0100
regfio: Improve handling of malformed registry hive files
* next_record: A malformed file can lead to an endless loop.
* regfio_rootkey: Supplying a malformed registry hive file to the
registry hive I/O code can lead to out-of-bounds reads.
Test cases are included. Both issues resolved have been identified using
AddressSanitizer.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13840
Signed-off-by: Michael Hanselmann <public at hansmi.ch>
Reviewed-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
(cherry picked from commit 601afd690346087fbd53819dba9b1afa81560064)
commit b5ae06cc65322bc60c6dd1277c309db20d2ec2b2
Author: Michael Hanselmann <public at hansmi.ch>
Date: Tue Mar 19 00:47:52 2019 +0100
regfio: Add trivial unit test
An upcoming commit will resolve two cases of insufficient handling of
mangled registry hive files and will include unit tests.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13840
Signed-off-by: Michael Hanselmann <public at hansmi.ch>
Reviewed-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
(cherry picked from commit 9b2cb845b23cd1c91ab3b5ea8ad791b18b3ab733)
commit 223352ee944695053d9c9ba0d41442a207ad8ee6
Author: Michael Hanselmann <public at hansmi.ch>
Date: Sun Mar 17 16:20:47 2019 +0100
regfio: Use correct function names in debug information
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13840
Signed-off-by: Michael Hanselmann <public at hansmi.ch>
Reviewed-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
(cherry picked from commit aa6b355858a0d8b77bf49384e5329642add1a5ff)
commit 4644b23b91c66a406d53ddcf49bedfe951185bf9
Author: Michael Hanselmann <public at hansmi.ch>
Date: Sun Mar 17 13:04:52 2019 +0100
Fix typos in "valid"
s/vald/valid/
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13840
Signed-off-by: Michael Hanselmann <public at hansmi.ch>
Reviewed-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
(cherry picked from commit 305346d360d3c13fd315c1af27b037f46fd10650)
commit 87ffad41af17ee07f8e6ba3d0c4c7cd1af8abd7a
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Fri Mar 22 15:24:47 2019 +1300
py/kcc_utils: py2.6 compatibility
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13837
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Garming Sam <garming at catalyst.net.nz>
commit d44f2157a729e2fa0614cf80f53b1bd7195d3547
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Wed Mar 20 12:02:09 2019 +1300
py/graph: use 2.6 compatible check for set membership
It is better this way anyway.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13837
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet at samba.org>
Autobuild-Date(master): Wed Mar 20 06:36:05 UTC 2019 on sn-devel-144
(cherry picked from commit c0aca17a4c9ec06f0127d5c972f3fa979a87a77f)
commit 42b62465fccee93824a31903de2d410ab121e152
Author: Stefan Metzmacher <metze at samba.org>
Date: Tue Mar 19 13:05:16 2019 +0100
dbcheck: use the str() value of the "name" attribute
We do the same with the rdn attribute value
and we need the same logic on both in order to
check they are the same.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13816
Signed-off-by: Stefan Metzmacher <metze at samba.org>
Reviewed-by: Noel Power <npower at samba.org>
(cherry picked from commit dd6f0dad218ec1d5aa38ea8aa6848ec81035cb3f)
commit 693c349874f66304ba41b0cca68275e5ecf8c07c
Author: Stefan Metzmacher <metze at samba.org>
Date: Tue Mar 12 11:41:01 2019 +0100
dbcheck: don't check expired tombstone objects by default anymore
These will be removed anyway and any change on them risks to
be an originating update that causes replication problems.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13816
Signed-off-by: Stefan Metzmacher <metze at samba.org>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet at samba.org>
Autobuild-Date(master): Thu Mar 14 03:12:27 UTC 2019 on sn-devel-144
(cherry picked from commit a2c5f8cf41c2dfdc4f122e8427d1dfeabb6ba311)
commit 3fca3dcc1c902250a6716aa4356f36cd9182ed1b
Author: Stefan Metzmacher <metze at samba.org>
Date: Tue Mar 12 11:38:22 2019 +0100
blackbox/dbcheck-links.sh: prepare regression test for skipping expired tombstones
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13816
Signed-off-by: Stefan Metzmacher <metze at samba.org>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
(cherry picked from commit b096a3117ed9249fd6f65f3221a26c88efbba3b8)
commit 543fc3e9c04f45b0bdac75daa1055fc4e7904637
Author: Stefan Metzmacher <metze at samba.org>
Date: Tue Mar 12 11:04:33 2019 +0100
blackbox/dbcheck*.sh: pass --selftest-check-expired-tombstones to dbcheck
These tests operate on provision dumps created long ago, they still
want to run tests on deleted objects, when the next commits remove
processing expired tombstone objects in dbcheck.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13816
Signed-off-by: Stefan Metzmacher <metze at samba.org>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
(cherry picked from commit 5fccc4e9044d2e57be33471f5e6b9be7cc37ac3a)
commit 02f3d0a1a2c8923c1403ebef5da903bbf586fc49
Author: Stefan Metzmacher <metze at samba.org>
Date: Tue Mar 12 11:02:18 2019 +0100
dbcheck: add --selftest-check-expired-tombstones cmdline option
This will be used by dbcheck tests which operate on static/old provision
dumps in the following commits.
Signed-off-by: Stefan Metzmacher <metze at samba.org>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
(cherry picked from commit 6f9c5ed8de47bb98e21e8064d8e90f963f2f71ca)
commit aebf46d957fa2d39c6575a79458d4a819a92ddda
Author: Stefan Metzmacher <metze at samba.org>
Date: Tue Mar 12 10:25:40 2019 +0100
python/samba/netcmd: provide SUPPRESS_HELP via Option class
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13816
Signed-off-by: Stefan Metzmacher <metze at samba.org>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
(cherry picked from commit b61d580fb7dba8ff94e9e98c958e324865cd2f1d)
commit 107883dff6cc42489169035dc36bfdf313cce4d7
Author: Stefan Metzmacher <metze at samba.org>
Date: Thu Feb 28 18:22:18 2019 +0100
dbcheck: detect the change after deletion bug
Old versions of 'samba-tool dbcheck' could reanimate
deleted objects, when running at the same time as the
tombstone garbage collection.
When the (deleted) parent of a deleted object
(with the DISALLOW_MOVE_ON_DELETE bit in systemFlags),
is removed before the object itself, dbcheck moved
it in the LostAndFound[Config] subtree of the partition
as an originating change. That means that the object
will be in tombstone state again for 180 days on the local
DC. And other DCs fail to replicate the object as
it's already removed completely there and the replication
only gives the name and lastKnownParent attributes, because
all other attributes should already be known to the other DC.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13816
Signed-off-by: Stefan Metzmacher <metze at samba.org>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
(cherry picked from commit a1658b306d85452407388b91a745078c9c1f7dc7)
commit 860b04aa7ae7ea700c7ff23bb08f6d6642cfa135
Author: Stefan Metzmacher <metze at samba.org>
Date: Mon Mar 11 23:14:02 2019 +0100
blackbox/dbcheck-links.sh: add regression test for lost deleted object repair
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13816
Signed-off-by: Stefan Metzmacher <metze at samba.org>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
(cherry picked from commit 1ccc21a34d295be3bb2ab481a5918003eae88bf4)
commit 45850169a9cfe6cd42df820a773c27bbee9a2448
Author: Stefan Metzmacher <metze at samba.org>
Date: Thu Feb 28 18:16:27 2019 +0100
dbcheck: add find_repl_attid() helper function
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13816
Signed-off-by: Stefan Metzmacher <metze at samba.org>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
(cherry picked from commit 598e38d2a5e0832429ba65b4e55bf7127618f894)
commit 7402d9cfcf77f188cef488c4d1f47b0c1900564b
Author: Stefan Metzmacher <metze at samba.org>
Date: Mon Feb 25 15:35:22 2019 +0100
dbcheck: don't remove dangling one-way links on already deleted objects
This would typically happen when the garbage collection
removed a parent object before a child object (both with
the DISALLOW_MOVE_ON_DELETE bit set in systemFlags),
while dbcheck is running at the same time as the garbage collection.
In this case the lastKnownParent attributes points a non existing
object.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13816
Signed-off-by: Stefan Metzmacher <metze at samba.org>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
(cherry picked from commit e388e599495b6d7c38b8b6966332e27f8b958783)
commit 07ebd654a0169d94d7471fa94bb813fe971f7ae8
Author: Stefan Metzmacher <metze at samba.org>
Date: Mon Feb 25 15:35:22 2019 +0100
dbcheck: don't move already deleted objects to LostAndFound
This would typically happen when the garbage collection
removed a parent object before a child object (both with
the DISALLOW_MOVE_ON_DELETE bit set in systemFlags),
while dbcheck is running at the same time as the garbage collection.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13816
Signed-off-by: Stefan Metzmacher <metze at samba.org>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
(cherry picked from commit 6d50ee74920c39cdb18b427bfaaf200775bf2d73)
commit 76de43f052f537b0d3f17f416728c008a65d3a75
Author: Stefan Metzmacher <metze at samba.org>
Date: Mon Feb 25 15:09:36 2019 +0100
dbcheck: do isDeleted, systemFlags and replPropertyMetaData detection first
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13816
Signed-off-by: Stefan Metzmacher <metze at samba.org>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
(cherry picked from commit 9afcd5331ce567bd80d35175f8e4e21c506e9347)
commit 0aaf7c98bb754c44e6d3cc303f713df24d7cfb90
Author: Stefan Metzmacher <metze at samba.org>
Date: Mon Mar 11 22:45:46 2019 +0100
dbcheck: use DSDB_CONTROL_DBCHECK_FIX_LINK_DN_NAME when renaming deleted objects
We should never do originating updates on deleted objects.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13816
Signed-off-by: Stefan Metzmacher <metze at samba.org>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
(cherry picked from commit 07a8326746f0c444eedf3860b178fc29d84e8d16)
commit 44c83b09c6005df63caa71acddcf1e4006392514
Author: Stefan Metzmacher <metze at samba.org>
Date: Mon Mar 11 22:38:38 2019 +0100
dsdb:repl_meta_data: allow CONTROL_DBCHECK_FIX_LINK_DN_NAME to by pass rename
We need a way to rename an object without updating the replication meta
data.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13816
Signed-off-by: Stefan Metzmacher <metze at samba.org>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
(cherry picked from commit 3e8a435d27da899d0e3dab7cbc0a1c738067eba3)
commit 9339b096793fee4b7deca21bce541999fa8230cf
Author: Stefan Metzmacher <metze at samba.org>
Date: Mon Mar 11 14:52:57 2019 +0100
blackbox/dbcheck-links.sh: reproduce lost deleted object problem
When a parent object is removed during the tombstone garbage collection
before a child object and samba-tool dbcheck runs at the same time, the
following can happen:
- If the object child had DISALLOW_MOVE_ON_DELETE in systemFlags,
samba-tool dbcheck moves the object under the LostAndFound[Config]
object (as an originating update!)
- The lastKnownParent attribute is removed (as an originating update!)
These originating updates cause the object to have an extended time
as tombstone. And these changes are replicated to other DCs,
which very likely already removed the object completely!
This means the destination DC of replication has no chance to handle
the object it gets from the source DC with just 2 attributes (name, lastKnownParent).
The destination logs something like:
No objectClass found in replPropertyMetaData
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13816
Signed-off-by: Stefan Metzmacher <metze at samba.org>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
(cherry picked from commit 5357f591accffbf8c62335c308b985811b66f0b5)
commit 7bcb072965211b98f3a94eaa4b60f9b0fe4c9348
Author: Stefan Metzmacher <metze at samba.org>
Date: Wed Feb 27 08:22:09 2019 +0100
selftest: force running with TZ=UTC
Signed-off-by: Stefan Metzmacher <metze at samba.org>
Reviewed-by: Andreas Schneider <asn at samba.org>
Autobuild-User(master): Stefan Metzmacher <metze at samba.org>
Autobuild-Date(master): Wed Feb 27 11:24:59 UTC 2019 on sn-devel-144
(cherry picked from commit 4f307f2302b0fe8fd0fc6379eb8e6491faf8520c)
commit 5602db1b1d5fa16c1d5578720ad48b8785f9f0a1
Author: Noel Power <noel.power at suse.com>
Date: Fri Sep 7 14:38:54 2018 +0100
python/samba: extra ndr_unpack needs bytes function
(cherry picked from commit 8db43696e70d7c4cb21172b7e7461cf6a72914a2)
commit 139da67cb3bd6baf8d2747e20fb0ff4886c2d6cb
Author: Noel Power <noel.power at suse.com>
Date: Fri Sep 7 12:42:19 2018 +0100
python/samba: PY3 port for ridalloc_exop test to work
Signed-off-by: Noel Power <noel.power at suse.com>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
(cherry picked from commit fc13a1268a4a9de94efd312a8309aa55d331ae19)
-----------------------------------------------------------------------
Summary of changes:
python/samba/dbchecker.py | 234 ++++++++++++--
python/samba/graph.py | 2 +-
python/samba/kcc/kcc_utils.py | 4 +-
python/samba/netcmd/__init__.py | 1 +
python/samba/netcmd/dbcheck.py | 15 +-
python/samba/remove_dc.py | 4 +-
selftest/selftest.pl | 3 +
selftest/tests.py | 2 +
source3/registry/regfio.c | 29 +-
source3/registry/tests/test_regfio.c | 184 +++++++++++
source3/torture/torture.c | 2 +-
source3/wscript | 3 +
source3/wscript_build | 6 +
source4/dsdb/samdb/ldb_modules/repl_meta_data.c | 7 +
...cted-dbcheck-link-output-lost-deleted-user1.txt | 9 +
...cted-dbcheck-link-output-lost-deleted-user2.txt | 8 +
...cted-dbcheck-link-output-lost-deleted-user3.txt | 19 ++
testdata/samba3/regfio_corrupt_hbin1.dat | Bin 0 -> 5120 bytes
testdata/samba3/regfio_corrupt_lf_subkeys.dat | Bin 0 -> 5120 bytes
testprogs/blackbox/dbcheck-links.sh | 342 ++++++++++++++++++++-
testprogs/blackbox/dbcheck-oldrelease.sh | 14 +-
21 files changed, 827 insertions(+), 61 deletions(-)
create mode 100644 source3/registry/tests/test_regfio.c
create mode 100644 source4/selftest/provisions/release-4-5-0-pre1/expected-dbcheck-link-output-lost-deleted-user1.txt
create mode 100644 source4/selftest/provisions/release-4-5-0-pre1/expected-dbcheck-link-output-lost-deleted-user2.txt
create mode 100644 source4/selftest/provisions/release-4-5-0-pre1/expected-dbcheck-link-output-lost-deleted-user3.txt
create mode 100644 testdata/samba3/regfio_corrupt_hbin1.dat
create mode 100644 testdata/samba3/regfio_corrupt_lf_subkeys.dat
Changeset truncated at 500 lines:
diff --git a/python/samba/dbchecker.py b/python/samba/dbchecker.py
index a35c4075a43..bd43667b99f 100644
--- a/python/samba/dbchecker.py
+++ b/python/samba/dbchecker.py
@@ -41,7 +41,8 @@ class dbcheck(object):
def __init__(self, samdb, samdb_schema=None, verbose=False, fix=False,
yes=False, quiet=False, in_transaction=False,
- reset_well_known_acls=False):
+ reset_well_known_acls=False,
+ check_expired_tombstones=False):
self.samdb = samdb
self.dict_oid_name = None
self.samdb_schema = (samdb_schema or samdb)
@@ -88,6 +89,8 @@ class dbcheck(object):
self.fix_doubled_userparameters = False
self.fix_sid_rid_set_conflict = False
self.reset_well_known_acls = reset_well_known_acls
+ self.check_expired_tombstones = check_expired_tombstones
+ self.expired_tombstones = 0
self.reset_all_well_known_acls = False
self.in_transaction = in_transaction
self.infrastructure_dn = ldb.Dn(samdb, "CN=Infrastructure," + samdb.domain_dn())
@@ -101,6 +104,7 @@ class dbcheck(object):
self.fix_missing_deleted_objects = False
self.fix_replica_locations = False
self.fix_missing_rid_set_master = False
+ self.fix_changes_after_deletion_bug = False
self.dn_set = set()
self.link_id_cache = {}
@@ -189,6 +193,14 @@ class dbcheck(object):
else:
self.rid_set_dn = None
+ ntds_service_dn = "CN=Directory Service,CN=Windows NT,CN=Services,%s" % \
+ self.samdb.get_config_basedn().get_linearized()
+ res = samdb.search(base=ntds_service_dn,
+ scope=ldb.SCOPE_BASE,
+ expression="(objectClass=nTDSService)",
+ attrs=["tombstoneLifetime"])
+ self.tombstoneLifetime = int(res[0]["tombstoneLifetime"][0])
+
self.compatibleFeatures = []
self.requiredFeatures = []
@@ -224,6 +236,13 @@ class dbcheck(object):
if DN is None:
error_count += self.check_rootdse()
+ if self.expired_tombstones > 0:
+ self.report("NOTICE: found %d expired tombstones, "
+ "'samba' will remove them daily, "
+ "'samba-tool domain tombstones expunge' "
+ "would do that immediately." % (
+ self.expired_tombstones))
+
if error_count != 0 and not self.fix:
self.report("Please use --fix to fix these errors")
@@ -549,6 +568,19 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base)))
def err_missing_target_dn_or_GUID(self, dn, attrname, val, dsdb_dn):
"""handle a missing target DN (if specified, GUID form can't be found,
and otherwise DN string form can't be found)"""
+
+ # Don't change anything if the object itself is deleted
+ if str(dn).find('\\0ADEL') != -1:
+ # We don't bump the error count as Samba produces these
+ # in normal operation
+ self.report("WARNING: no target object found for GUID "
+ "component link %s in deleted object "
+ "%s - %s" % (attrname, dn, val))
+ self.report("Not removing dangling one-way "
+ "link on deleted object "
+ "(tombstone garbage collection in progress?)")
+ return 0
+
# check if its a backlink
linkID, _ = self.get_attr_linkID_and_reverse_name(attrname)
if (linkID & 1 == 0) and str(dsdb_dn).find('\\0ADEL') == -1:
@@ -859,7 +891,7 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base)))
else:
self.samdb.transaction_cancel()
- def err_wrong_dn(self, obj, new_dn, rdn_attr, rdn_val, name_val):
+ def err_wrong_dn(self, obj, new_dn, rdn_attr, rdn_val, name_val, controls):
'''handle a wrong dn'''
new_rdn = ldb.Dn(self.samdb, str(new_dn))
@@ -876,7 +908,7 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base)))
self.report("Not renaming %s to %s" % (obj.dn, new_dn))
return
- if self.do_rename(obj.dn, new_rdn, new_parent, ["show_recycled:1", "relax:0"],
+ if self.do_rename(obj.dn, new_rdn, new_parent, controls,
"Failed to rename object %s into %s" % (obj.dn, new_dn)):
self.report("Renamed %s into %s" % (obj.dn, new_dn))
@@ -1450,6 +1482,12 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base)))
return error_count
+ def find_repl_attid(self, repl, attid):
+ for o in repl.ctr.array:
+ if o.attid == attid:
+ return o
+
+ return None
def get_originating_time(self, val, attid):
'''Read metadata properties and return the originating time for
@@ -1459,12 +1497,9 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base)))
'''
repl = ndr_unpack(drsblobs.replPropertyMetaDataBlob, str(val))
- obj = repl.ctr
-
- for o in repl.ctr.array:
- if o.attid == attid:
- return o.originating_change_time
-
+ o = self.find_repl_attid(repl, attid)
+ if o is not None:
+ return o.originating_change_time
return 0
def process_metadata(self, dn, val):
@@ -1717,6 +1752,131 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base)))
self.report("Fixed attribute '%s' of '%s'\n" % (sd_attr, dn))
self.samdb.set_session_info(self.system_session_info)
+ def is_expired_tombstone(self, dn, repl_val):
+ if self.check_expired_tombstones:
+ # This is not the default, it's just
+ # used to keep dbcheck tests work with
+ # old static provision dumps
+ return False
+
+ repl = ndr_unpack(drsblobs.replPropertyMetaDataBlob, repl_val)
+
+ isDeleted = self.find_repl_attid(repl, drsuapi.DRSUAPI_ATTID_isDeleted)
+
+ delete_time = samba.nttime2unix(isDeleted.originating_change_time)
+ current_time = time.time()
+
+ tombstone_delta = self.tombstoneLifetime * (24 * 60 * 60)
+
+ delta = current_time - delete_time
+ if delta <= tombstone_delta:
+ return False
+
+ self.report("SKIPING: object %s is an expired tombstone" % dn)
+ self.report("isDeleted: attid=0x%08x version=%d invocation=%s usn=%s (local=%s) at %s" % (
+ isDeleted.attid,
+ isDeleted.version,
+ isDeleted.originating_invocation_id,
+ isDeleted.originating_usn,
+ isDeleted.local_usn,
+ time.ctime(samba.nttime2unix(isDeleted.originating_change_time))))
+ self.expired_tombstones += 1
+ return True
+
+ def find_changes_after_deletion(self, repl_val):
+ repl = ndr_unpack(drsblobs.replPropertyMetaDataBlob, repl_val)
+
+ isDeleted = self.find_repl_attid(repl, drsuapi.DRSUAPI_ATTID_isDeleted)
+
+ delete_time = samba.nttime2unix(isDeleted.originating_change_time)
+
+ tombstone_delta = self.tombstoneLifetime * (24 * 60 * 60)
+
+ found = []
+ for o in repl.ctr.array:
+ if o.attid == drsuapi.DRSUAPI_ATTID_isDeleted:
+ continue
+
+ if o.local_usn <= isDeleted.local_usn:
+ continue
+
+ if o.originating_change_time <= isDeleted.originating_change_time:
+ continue
+
+ change_time = samba.nttime2unix(o.originating_change_time)
+
+ delta = change_time - delete_time
+ if delta <= tombstone_delta:
+ continue
+
+ # If the modification happened after the tombstone lifetime
+ # has passed, we have a bug as the object might be deleted
+ # already on other DCs and won't be able to replicate
+ # back
+ found.append(o)
+
+ return found, isDeleted
+
+ def has_changes_after_deletion(self, dn, repl_val):
+ found, isDeleted = self.find_changes_after_deletion(repl_val)
+ if len(found) == 0:
+ return False
+
+ def report_attid(o):
+ try:
+ attname = self.samdb_schema.get_lDAPDisplayName_by_attid(o.attid)
+ except KeyError:
+ attname = "<unknown:0x%x08x>" % o.attid
+
+ self.report("%s: attid=0x%08x version=%d invocation=%s usn=%s (local=%s) at %s" % (
+ attname, o.attid, o.version,
+ o.originating_invocation_id,
+ o.originating_usn,
+ o.local_usn,
+ time.ctime(samba.nttime2unix(o.originating_change_time))))
+
+ self.report("ERROR: object %s, has changes after deletion" % dn)
+ report_attid(isDeleted)
+ for o in found:
+ report_attid(o)
+
+ return True
+
+ def err_changes_after_deletion(self, dn, repl_val):
+ found, isDeleted = self.find_changes_after_deletion(repl_val)
+
+ in_schema_nc = dn.is_child_of(self.schema_dn)
+ rdn_attr = dn.get_rdn_name()
+ rdn_attid = self.samdb_schema.get_attid_from_lDAPDisplayName(rdn_attr,
+ is_schema_nc=in_schema_nc)
+
+ unexpected = []
+ for o in found:
+ if o.attid == rdn_attid:
+ continue
+ if o.attid == drsuapi.DRSUAPI_ATTID_name:
+ continue
+ if o.attid == drsuapi.DRSUAPI_ATTID_lastKnownParent:
+ continue
+ try:
+ attname = self.samdb_schema.get_lDAPDisplayName_by_attid(o.attid)
+ except KeyError:
+ attname = "<unknown:0x%x08x>" % o.attid
+ unexpected.append(attname)
+
+ if len(unexpected) > 0:
+ self.report('Unexpeted attributes: %s' % ",".join(unexpected))
+ self.report('Not fixing changes after deletion bug')
+ return
+
+ if not self.confirm_all('Delete broken tombstone object %s deleted %s days ago?' % (
+ dn, self.tombstoneLifetime), 'fix_changes_after_deletion_bug'):
+ self.report('Not fixing changes after deletion bug')
+ return
+
+ if self.do_delete(dn, ["relax:0"],
+ "Failed to remove DN %s" % dn):
+ self.report("Removed DN %s" % dn)
def has_replmetadata_zero_invocationid(self, dn, repl_meta_data):
repl = ndr_unpack(drsblobs.replPropertyMetaDataBlob,
@@ -1989,8 +2149,7 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base)))
raise
else:
instancetype |= dsdb.INSTANCE_TYPE_NC_ABOVE
-
- if self.write_ncs is not None and str(nc_root) in self.write_ncs:
+ if self.write_ncs is not None and str(nc_root) in [str(x) for x in self.write_ncs]:
instancetype |= dsdb.INSTANCE_TYPE_WRITE
return instancetype
@@ -2070,7 +2229,6 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base)))
error_count = 0
set_attrs_from_md = set()
set_attrs_seen = set()
- got_repl_property_meta_data = False
got_objectclass = False
nc_dn = self.samdb.get_nc_root(obj.dn)
@@ -2088,6 +2246,26 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base)))
name_val = None
isDeleted = False
systemFlags = 0
+ repl_meta_data_val = None
+
+ for attrname in obj:
+ if str(attrname).lower() == 'isdeleted':
+ if str(obj[attrname][0]) != "FALSE":
+ isDeleted = True
+
+ if str(attrname).lower() == 'systemflags':
+ systemFlags = int(obj[attrname][0])
+
+ if str(attrname).lower() == 'replpropertymetadata':
+ repl_meta_data_val = obj[attrname][0]
+
+ if isDeleted and repl_meta_data_val:
+ if self.has_changes_after_deletion(dn, repl_meta_data_val):
+ error_count += 1
+ self.err_changes_after_deletion(dn, repl_meta_data_val)
+ return error_count
+ if self.is_expired_tombstone(dn, repl_meta_data_val):
+ return error_count
for attrname in obj:
if attrname == 'dn' or attrname == "distinguishedName":
@@ -2102,7 +2280,7 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base)))
self.report("ERROR: Not fixing num_values(%d) for '%s' on '%s'" %
(len(obj[attrname]), attrname, str(obj.dn)))
else:
- name_val = obj[attrname][0]
+ name_val = str(obj[attrname][0])
if str(attrname).lower() == str(obj.dn.get_rdn_name()).lower():
object_rdn_attr = attrname
@@ -2111,14 +2289,7 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base)))
self.report("ERROR: Not fixing num_values(%d) for '%s' on '%s'" %
(len(obj[attrname]), attrname, str(obj.dn)))
else:
- object_rdn_val = obj[attrname][0]
-
- if str(attrname).lower() == 'isdeleted':
- if obj[attrname][0] != "FALSE":
- isDeleted = True
-
- if str(attrname).lower() == 'systemflags':
- systemFlags = int(obj[attrname][0])
+ object_rdn_val = str(obj[attrname][0])
if str(attrname).lower() == 'replpropertymetadata':
if self.has_replmetadata_zero_invocationid(dn, obj[attrname]):
@@ -2149,7 +2320,6 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base)))
self.report("ERROR: Not fixing incorrect inital attributeID in '%s' on '%s', it should be objectClass" %
(attrname, str(dn)))
- got_repl_property_meta_data = True
continue
if str(attrname).lower() == 'ntsecuritydescriptor':
@@ -2289,7 +2459,7 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base)))
if str(attrname).lower() == "instancetype":
calculated_instancetype = self.calculate_instancetype(dn)
- if len(obj["instanceType"]) != 1 or obj["instanceType"][0] != str(calculated_instancetype):
+ if len(obj["instanceType"]) != 1 or int(obj["instanceType"][0]) != calculated_instancetype:
error_count += 1
self.err_wrong_instancetype(obj, calculated_instancetype)
@@ -2307,9 +2477,11 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base)))
if name_val is not None:
parent_dn = None
+ controls = ["show_recycled:1", "relax:0"]
if isDeleted:
if not (systemFlags & samba.dsdb.SYSTEM_FLAG_DISALLOW_MOVE_ON_DELETE):
parent_dn = deleted_objects_dn
+ controls += ["local_oid:%s:1" % dsdb.DSDB_CONTROL_DBCHECK_FIX_LINK_DN_NAME]
if parent_dn is None:
parent_dn = obj.dn.parent()
expected_dn = ldb.Dn(self.samdb, "RDN=RDN,%s" % (parent_dn))
@@ -2320,19 +2492,20 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base)))
if expected_dn != obj.dn:
error_count += 1
- self.err_wrong_dn(obj, expected_dn, object_rdn_attr, object_rdn_val, name_val)
+ self.err_wrong_dn(obj, expected_dn, object_rdn_attr,
+ object_rdn_val, name_val, controls)
elif obj.dn.get_rdn_value() != object_rdn_val:
error_count += 1
self.report("ERROR: Not fixing %s=%r on '%s'" % (object_rdn_attr, object_rdn_val, str(obj.dn)))
show_dn = True
- if got_repl_property_meta_data:
+ if repl_meta_data_val:
if obj.dn == deleted_objects_dn:
isDeletedAttId = 131120
# It's 29/12/9999 at 23:59:59 UTC as specified in MS-ADTS 7.1.1.4.2 Deleted Objects Container
expectedTimeDo = 2650466015990000000
- originating = self.get_originating_time(obj["replPropertyMetaData"], isDeletedAttId)
+ originating = self.get_originating_time(repl_meta_data_val, isDeletedAttId)
if originating != expectedTimeDo:
if self.confirm_all("Fix isDeleted originating_change_time on '%s'" % str(dn), 'fix_time_metadata'):
nmsg = ldb.Message()
@@ -2367,8 +2540,13 @@ newSuperior: %s""" % (str(from_dn), str(to_rdn), str(to_base)))
except ldb.LdbError as e11:
(enum, estr) = e11.args
if enum == ldb.ERR_NO_SUCH_OBJECT:
- self.err_missing_parent(obj)
- error_count += 1
+ if isDeleted:
+ self.report("WARNING: parent object not found for %s" % (obj.dn))
+ self.report("Not moving to LostAndFound "
+ "(tombstone garbage collection in progress?)")
+ else:
+ self.err_missing_parent(obj)
+ error_count += 1
else:
raise
diff --git a/python/samba/graph.py b/python/samba/graph.py
index e80a8820667..25673ac6770 100644
--- a/python/samba/graph.py
+++ b/python/samba/graph.py
@@ -88,7 +88,7 @@ def shorten_vertex_names(vertices, suffix=',...', aggressive=False):
try:
while True:
c = set(x[i] for x in vlist)
- if len(c) > 1 or c == {'*'}:
+ if len(c) > 1 or '*' in c:
break
i -= 1
except IndexError:
diff --git a/python/samba/kcc/kcc_utils.py b/python/samba/kcc/kcc_utils.py
index 1457ea355e0..0fbf55939ae 100644
--- a/python/samba/kcc/kcc_utils.py
+++ b/python/samba/kcc/kcc_utils.py
@@ -31,7 +31,7 @@ from samba.dcerpc import (
)
from samba.common import dsdb_Dn
from samba.ndr import ndr_unpack, ndr_pack
-from collections import Counter
+from collections import defaultdict
class KCCError(Exception):
@@ -2287,7 +2287,7 @@ def uncovered_sites_to_cover(samdb, site_name):
scope=ldb.SCOPE_SUBTREE,
expression="(objectClass=site)")
- sites_in_use = Counter()
+ sites_in_use = defaultdict(int)
dc_count = 0
# Assume server is of form DC,Servers,Site-ABCD because of schema
diff --git a/python/samba/netcmd/__init__.py b/python/samba/netcmd/__init__.py
index 9b144d97f43..8032f05ea28 100644
--- a/python/samba/netcmd/__init__.py
+++ b/python/samba/netcmd/__init__.py
@@ -24,6 +24,7 @@ import sys, traceback
import textwrap
class Option(optparse.Option):
+ SUPPRESS_HELP = optparse.SUPPRESS_HELP
pass
# This help formatter does text wrapping and preserves newlines
diff --git a/python/samba/netcmd/dbcheck.py b/python/samba/netcmd/dbcheck.py
index 210d4a26a1d..12a7bbe2cf1 100644
--- a/python/samba/netcmd/dbcheck.py
+++ b/python/samba/netcmd/dbcheck.py
@@ -74,13 +74,18 @@ class cmd_dbcheck(Command):
Option("--reset-well-known-acls", dest="reset_well_known_acls", default=False, action="store_true", help="reset ACLs on objects with well known default ACL values to the default"),
Option("-H", "--URL", help="LDB URL for database or target server (defaults to local SAM database)",
type=str, metavar="URL", dest="H"),
- ]
+ Option("--selftest-check-expired-tombstones",
+ dest="selftest_check_expired_tombstones", default=False, action="store_true",
+ help=Option.SUPPRESS_HELP), # This is only used by tests
+ ]
def run(self, DN=None, H=None, verbose=False, fix=False, yes=False,
cross_ncs=False, quiet=False,
scope="SUB", credopts=None, sambaopts=None, versionopts=None,
attrs=None, reindex=False, force_modules=False,
- reset_well_known_acls=False, yes_rules=[]):
+ reset_well_known_acls=False,
+ selftest_check_expired_tombstones=False,
+ yes_rules=[]):
lp = sambaopts.get_loadparm()
@@ -131,8 +136,10 @@ class cmd_dbcheck(Command):
started_transaction = True
try:
chk = dbcheck(samdb, samdb_schema=samdb_schema, verbose=verbose,
- fix=fix, yes=yes, quiet=quiet, in_transaction=started_transaction,
- reset_well_known_acls=reset_well_known_acls)
+ fix=fix, yes=yes, quiet=quiet,
+ in_transaction=started_transaction,
+ reset_well_known_acls=reset_well_known_acls,
+ check_expired_tombstones=selftest_check_expired_tombstones)
for option in yes_rules:
if hasattr(chk, option):
diff --git a/python/samba/remove_dc.py b/python/samba/remove_dc.py
index e6513ae04fa..b6cde067502 100644
--- a/python/samba/remove_dc.py
+++ b/python/samba/remove_dc.py
@@ -236,7 +236,7 @@ def offline_remove_server(samdb, logger,
computer_dn = None
try:
- dnsHostName = msgs[0]["dnsHostName"][0]
+ dnsHostName = str(msgs[0]["dnsHostName"][0])
except KeyError:
dnsHostName = None
@@ -267,7 +267,7 @@ def offline_remove_server(samdb, logger,
samdb.delete(computer_dn, ["tree_delete:0"])
if "dnsHostName" in msgs[0]:
--
Samba Shared Repository
More information about the samba-cvs
mailing list