[PATCH] Improve replication with large databases
Tim Beale
timbeale at catalyst.net.nz
Sun Oct 14 20:33:22 UTC 2018
Here's a couple of minor improvements when replicating (i.e. join or
clone/backup) a DB with 10,00-25,000 users.
1. Large amount of memory consumed by the python code while replicating
a partition was in progress. This wasn't a memory leak - the memory
would be freed once the entire partition had finished replicating.
However, while replication was in progress, it seemed to accumulate
memory for every reply chunk of replication data it received. With a
large DB, this could have significant memory overhead.
2. Improve debug so it's easier to see how far through replicating the
links you are.
CI pass: https://gitlab.com/catalyst-samba/samba/pipelines/32738642
Review appreciated. Thanks.
-------------- next part --------------
From be47c1c62938b1466beb96bd72ec35ad41572d52 Mon Sep 17 00:00:00 2001
From: Tim Beale <timbeale at catalyst.net.nz>
Date: Fri, 12 Oct 2018 13:29:32 +1300
Subject: [PATCH 1/2] libnet/drs: Update replication debug to report link
progress
Update the replication debug (for joins/backups) so that it's easier to
see how far through syncing the links we are. E.g. with 150,000 links,
you just get screeds of debug like this, with no real idea how far
through the replication is.
Partition[DC=addom,DC=samba,DC=example,DC=com] objects[11816/11720]
linked_values[1500/150024]
Partition[DC=addom,DC=samba,DC=example,DC=com] objects[11816/11720]
linked_values[1500/150024]
Partition[DC=addom,DC=samba,DC=example,DC=com] objects[11816/11720]
linked_values[1500/150024]
This patch now applies to links the same debug logic we use for objects,
and changes it to look like:
Partition[DC=addom,DC=samba,DC=example,DC=com] objects[11816/11720]
linked_values[57024/150024]
Partition[DC=addom,DC=samba,DC=example,DC=com] objects[11816/11720]
linked_values[58524/150024]
Partition[DC=addom,DC=samba,DC=example,DC=com] objects[11816/11720]
linked_values[60024/150024]
Signed-off-by: Tim Beale <timbeale at catalyst.net.nz>
---
source4/libnet/libnet_vampire.c | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/source4/libnet/libnet_vampire.c b/source4/libnet/libnet_vampire.c
index c4ef608..5bd8df1 100644
--- a/source4/libnet/libnet_vampire.c
+++ b/source4/libnet/libnet_vampire.c
@@ -88,6 +88,7 @@ struct libnet_vampire_cb_state {
struct loadparm_context *lp_ctx;
struct tevent_context *event_ctx;
unsigned total_objects;
+ unsigned total_links;
char *last_partition;
const char *server_dn_str;
};
@@ -689,10 +690,12 @@ WERROR libnet_vampire_cb_store_chunk(void *private_data,
/* we want to show a count per partition */
if (!s->last_partition || strcmp(s->last_partition, c->partition->nc.dn) != 0) {
s->total_objects = 0;
+ s->total_links = 0;
talloc_free(s->last_partition);
s->last_partition = talloc_strdup(s, c->partition->nc.dn);
}
s->total_objects += object_count;
+ s->total_links += linked_attributes_count;
partition_dn = ldb_dn_new(s_dsa, s->ldb, c->partition->nc.dn);
if (partition_dn == NULL) {
@@ -705,7 +708,7 @@ WERROR libnet_vampire_cb_store_chunk(void *private_data,
if (nc_object_count) {
DEBUG(0,("Exop on[%s] objects[%u/%u] linked_values[%u/%u]\n",
c->partition->nc.dn, s->total_objects, nc_object_count,
- linked_attributes_count, nc_linked_attributes_count));
+ s->total_links, nc_linked_attributes_count));
} else {
DEBUG(0,("Exop on[%s] objects[%u] linked_values[%u]\n",
c->partition->nc.dn, s->total_objects, linked_attributes_count));
@@ -721,10 +724,10 @@ WERROR libnet_vampire_cb_store_chunk(void *private_data,
if (nc_object_count) {
DEBUG(0,("Partition[%s] objects[%u/%u] linked_values[%u/%u]\n",
c->partition->nc.dn, s->total_objects, nc_object_count,
- linked_attributes_count, nc_linked_attributes_count));
+ s->total_links, nc_linked_attributes_count));
} else {
DEBUG(0,("Partition[%s] objects[%u] linked_values[%u]\n",
- c->partition->nc.dn, s->total_objects, linked_attributes_count));
+ c->partition->nc.dn, s->total_objects, s->total_links));
}
nc_root = partition_dn;
}
--
2.7.4
From 529a44b4cb52b2d2f9cc7afac9ca405678651bbf Mon Sep 17 00:00:00 2001
From: Tim Beale <timbeale at catalyst.net.nz>
Date: Fri, 12 Oct 2018 13:54:34 +1300
Subject: [PATCH 2/2] drs_util: Improve memory usage when joining large DB
drs_Replicate.replicate() could consume a large amount of memory when
replicating a large DB. This is not a leak - the memory gets freed when
the function returns (i.e. once the partition is fully replicated).
However, while the partition is in the process of being replicated, it
accumulates memory for each replication chunk it receives. This can have
considerable overhead with 1000s of objects/links in the partition.
This was exhausting memory when joining a VM with 1Gb RAM to a DC with
25K users (average ~15 group memberships per user).
It seems that by storing a reference to something that's on the ctr's
talloc tree, it doesn't free up the memory for each ctr message (until
the function actually returns and req is destroyed).
With 10K users (and average 15 group memberships per user), .replicate()
consumed 211Mb of memory, according to talloc.report_full(). With this
patch, it goes down to just the current ctr message (1-2Mb).
Signed-off-by: Tim Beale <timbeale at catalyst.net.nz>
---
python/samba/drs_utils.py | 14 +++++++++++++-
1 file changed, 13 insertions(+), 1 deletion(-)
diff --git a/python/samba/drs_utils.py b/python/samba/drs_utils.py
index 05a9df7..dcd4d73 100644
--- a/python/samba/drs_utils.py
+++ b/python/samba/drs_utils.py
@@ -189,6 +189,16 @@ def drs_get_rodc_partial_attribute_set(samdb):
return partial_attribute_set
+def drs_copy_highwater_mark(hwm, new_hwm):
+ """
+ Copies the highwater mark by value, rather than by object reference. (This
+ avoids lingering talloc references to old GetNCChanges reply messages).
+ """
+ hwm.tmp_highest_usn = new_hwm.tmp_highest_usn
+ hwm.reserved_usn = new_hwm.reserved_usn
+ hwm.highest_usn = new_hwm.highest_usn
+
+
class drs_Replicate(object):
'''DRS replication calls'''
@@ -353,7 +363,9 @@ class drs_Replicate(object):
if ctr.more_data == 0:
break
- req.highwatermark = ctr.new_highwatermark
+
+ # update the request's HWM so we get the next chunk
+ drs_copy_highwater_mark(req.highwatermark, ctr.new_highwatermark)
return (num_objects, num_links)
--
2.7.4
More information about the samba-technical
mailing list