[SCM] Samba Shared Repository - branch master updated
Andrew Bartlett
abartlet at samba.org
Wed Oct 17 06:57:02 UTC 2018
The branch, master has been updated
via fdb6b86 drs_util: Improve memory usage when joining large DB
via 539daef libnet/drs: Update replication debug to report link progress
via 3713905 dns: dlz_bind9 reference count logging
from 2557ae5 lib: Move the "expired" for gencache_parse calculation into gencache.c
https://git.samba.org/?p=samba.git;a=shortlog;h=master
- Log -----------------------------------------------------------------
commit fdb6b86b8e326e1d9d1ca4be780ff374b6dddde2
Author: Tim Beale <timbeale at catalyst.net.nz>
Date: Fri Oct 12 13:54:34 2018 +1300
drs_util: Improve memory usage when joining large DB
drs_Replicate.replicate() could consume a large amount of memory when
replicating a large DB. This is not a leak - the memory gets freed when
the function returns (i.e. once the partition is fully replicated).
However, while the partition is in the process of being replicated, it
accumulates memory for each replication chunk it receives. This can have
considerable overhead with 1000s of objects/links in the partition.
This was exhausting memory when joining a VM with 1Gb RAM to a DC with
25K users (average ~15 group memberships per user).
It seems that by storing a reference to something that's on the ctr's
talloc tree, it doesn't free up the memory for each ctr message (until
the function actually returns and req is destroyed).
With 10K users (and average 15 group memberships per user), .replicate()
consumed 211Mb of memory, according to talloc.report_full(). With this
patch, it goes down to just the current ctr message (1-2Mb).
Signed-off-by: Tim Beale <timbeale at catalyst.net.nz>
Reviewed-by: Garming Sam <garming at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
Autobuild-User(master): Andrew Bartlett <abartlet at samba.org>
Autobuild-Date(master): Wed Oct 17 08:56:42 CEST 2018 on sn-devel-144
commit 539daefaf3f7edb6a630f3d57fb590f57a6ab688
Author: Tim Beale <timbeale at catalyst.net.nz>
Date: Fri Oct 12 13:29:32 2018 +1300
libnet/drs: Update replication debug to report link progress
Update the replication debug (for joins/backups) so that it's easier to
see how far through syncing the links we are. E.g. with 150,000 links,
you just get screeds of debug like this, with no real idea how far
through the replication is.
Partition[DC=addom,DC=samba,DC=example,DC=com] objects[11816/11720]
linked_values[1500/150024]
Partition[DC=addom,DC=samba,DC=example,DC=com] objects[11816/11720]
linked_values[1500/150024]
Partition[DC=addom,DC=samba,DC=example,DC=com] objects[11816/11720]
linked_values[1500/150024]
This patch now applies to links the same debug logic we use for objects,
and changes it to look like:
Partition[DC=addom,DC=samba,DC=example,DC=com] objects[11816/11720]
linked_values[57024/150024]
Partition[DC=addom,DC=samba,DC=example,DC=com] objects[11816/11720]
linked_values[58524/150024]
Partition[DC=addom,DC=samba,DC=example,DC=com] objects[11816/11720]
linked_values[60024/150024]
Signed-off-by: Tim Beale <timbeale at catalyst.net.nz>
Reviewed-by: Garming Sam <garming at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 3713905f8b4baa56b1bf571f71009cd77ccda6eb
Author: Aaron Haslett <aaronhaslett at catalyst.net.nz>
Date: Mon Oct 15 16:52:40 2018 +1300
dns: dlz_bind9 reference count logging
dlz_bind9 has to count the number of times the plugin is 'created' by bind's
plugin manager so it doesn't repeat setup. Logging doesn't reflect this
reference counting logic properly and so messages like "samba_dlz: shutdown"
can, confusingly, come up when the database connection has not actually been
severed. This patch adds the necessary logging.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13655
Signed-off-by: Aaron Haslett <aaronhaslett at catalyst.net.nz>
Reviewed-by: Jeremy Allison <jra at samba.org>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
-----------------------------------------------------------------------
Summary of changes:
python/samba/drs_utils.py | 14 +++++++++++++-
source4/dns_server/dlz_bind9.c | 13 ++++++++++++-
source4/libnet/libnet_vampire.c | 9 ++++++---
3 files changed, 31 insertions(+), 5 deletions(-)
Changeset truncated at 500 lines:
diff --git a/python/samba/drs_utils.py b/python/samba/drs_utils.py
index 05a9df7..dcd4d73 100644
--- a/python/samba/drs_utils.py
+++ b/python/samba/drs_utils.py
@@ -189,6 +189,16 @@ def drs_get_rodc_partial_attribute_set(samdb):
return partial_attribute_set
+def drs_copy_highwater_mark(hwm, new_hwm):
+ """
+ Copies the highwater mark by value, rather than by object reference. (This
+ avoids lingering talloc references to old GetNCChanges reply messages).
+ """
+ hwm.tmp_highest_usn = new_hwm.tmp_highest_usn
+ hwm.reserved_usn = new_hwm.reserved_usn
+ hwm.highest_usn = new_hwm.highest_usn
+
+
class drs_Replicate(object):
'''DRS replication calls'''
@@ -353,7 +363,9 @@ class drs_Replicate(object):
if ctr.more_data == 0:
break
- req.highwatermark = ctr.new_highwatermark
+
+ # update the request's HWM so we get the next chunk
+ drs_copy_highwater_mark(req.highwatermark, ctr.new_highwatermark)
return (num_objects, num_links)
diff --git a/source4/dns_server/dlz_bind9.c b/source4/dns_server/dlz_bind9.c
index 054f13e..43f3e57 100644
--- a/source4/dns_server/dlz_bind9.c
+++ b/source4/dns_server/dlz_bind9.c
@@ -618,6 +618,9 @@ _PUBLIC_ isc_result_t dlz_create(const char *dlzname,
char *errstring = NULL;
if (dlz_bind9_state != NULL) {
+ dlz_bind9_state->log(ISC_LOG_ERROR,
+ "samba_dlz: dlz_create ignored, #refs=%d",
+ dlz_bind9_state_ref_count);
*dbdata = dlz_bind9_state;
dlz_bind9_state_ref_count++;
return ISC_R_SUCCESS;
@@ -743,6 +746,10 @@ _PUBLIC_ isc_result_t dlz_create(const char *dlzname,
return ISC_R_SUCCESS;
failed:
+ state->log(ISC_LOG_INFO,
+ "samba_dlz: FAILED dlz_create call result=%d #refs=%d",
+ result,
+ dlz_bind9_state_ref_count);
talloc_free(state);
return result;
}
@@ -753,13 +760,17 @@ failed:
_PUBLIC_ void dlz_destroy(void *dbdata)
{
struct dlz_bind9_data *state = talloc_get_type_abort(dbdata, struct dlz_bind9_data);
- state->log(ISC_LOG_INFO, "samba_dlz: shutting down");
dlz_bind9_state_ref_count--;
if (dlz_bind9_state_ref_count == 0) {
+ state->log(ISC_LOG_INFO, "samba_dlz: shutting down");
talloc_unlink(state, state->samdb);
talloc_free(state);
dlz_bind9_state = NULL;
+ } else {
+ state->log(ISC_LOG_INFO,
+ "samba_dlz: dlz_destroy called. %d refs remaining.",
+ dlz_bind9_state_ref_count);
}
}
diff --git a/source4/libnet/libnet_vampire.c b/source4/libnet/libnet_vampire.c
index c4ef608..5bd8df1 100644
--- a/source4/libnet/libnet_vampire.c
+++ b/source4/libnet/libnet_vampire.c
@@ -88,6 +88,7 @@ struct libnet_vampire_cb_state {
struct loadparm_context *lp_ctx;
struct tevent_context *event_ctx;
unsigned total_objects;
+ unsigned total_links;
char *last_partition;
const char *server_dn_str;
};
@@ -689,10 +690,12 @@ WERROR libnet_vampire_cb_store_chunk(void *private_data,
/* we want to show a count per partition */
if (!s->last_partition || strcmp(s->last_partition, c->partition->nc.dn) != 0) {
s->total_objects = 0;
+ s->total_links = 0;
talloc_free(s->last_partition);
s->last_partition = talloc_strdup(s, c->partition->nc.dn);
}
s->total_objects += object_count;
+ s->total_links += linked_attributes_count;
partition_dn = ldb_dn_new(s_dsa, s->ldb, c->partition->nc.dn);
if (partition_dn == NULL) {
@@ -705,7 +708,7 @@ WERROR libnet_vampire_cb_store_chunk(void *private_data,
if (nc_object_count) {
DEBUG(0,("Exop on[%s] objects[%u/%u] linked_values[%u/%u]\n",
c->partition->nc.dn, s->total_objects, nc_object_count,
- linked_attributes_count, nc_linked_attributes_count));
+ s->total_links, nc_linked_attributes_count));
} else {
DEBUG(0,("Exop on[%s] objects[%u] linked_values[%u]\n",
c->partition->nc.dn, s->total_objects, linked_attributes_count));
@@ -721,10 +724,10 @@ WERROR libnet_vampire_cb_store_chunk(void *private_data,
if (nc_object_count) {
DEBUG(0,("Partition[%s] objects[%u/%u] linked_values[%u/%u]\n",
c->partition->nc.dn, s->total_objects, nc_object_count,
- linked_attributes_count, nc_linked_attributes_count));
+ s->total_links, nc_linked_attributes_count));
} else {
DEBUG(0,("Partition[%s] objects[%u] linked_values[%u]\n",
- c->partition->nc.dn, s->total_objects, linked_attributes_count));
+ c->partition->nc.dn, s->total_objects, s->total_links));
}
nc_root = partition_dn;
}
--
Samba Shared Repository
More information about the samba-cvs
mailing list