[PATCH] Create a custom testenv from backup-file for manual testing

Tim Beale timbeale at catalyst.net.nz
Thu Nov 1 04:36:36 UTC 2018


Updated the patch to automatically detect the realm/domain to use from
the backup file. Also removed some redundant selftest code.

CI link: https://gitlab.com/catalyst-samba/samba/pipelines/35044262

On 31/10/18 9:52 AM, Tim Beale wrote:
> The attached patch allows you to easily spin up any custom testenv that
> you want, for manual testing. All you need is a backup-file that the
> testenv DC will be based off.
>
> The main use-case is any testing that involves a large database.
> Creating user accounts is slow. Even with the improvements to the
> traffic_replay user generation, creating 5,000 users still takes close
> to 10 minutes. Instead of creating a blank testenv and slowly populating
> it with users, you can do this step once, take a backup, and then spin
> up the backup database multiple times as a new testenv.
>
> This testenv might be useful in other situations too. E.g. testing
> migrating databases across samba versions, you spot a DB corruption and
> want to try diagnosing/fixing it without completely losing the original
> problem state, you want to create a labdc-replica testenv of a real
> network, etc.
>
> Also fixed a couple of minor dnsupdate problems that were making
> debugging testenv bootstrapping issues difficult.
>
> CI pass: https://gitlab.com/catalyst-samba/samba/pipelines/34756736
>
-------------- next part --------------
From 2345e4202bd5185f26cba03d5c8e6f913b5b4c42 Mon Sep 17 00:00:00 2001
From: Tim Beale <timbeale at catalyst.net.nz>
Date: Fri, 26 Oct 2018 11:08:54 +1300
Subject: [PATCH 1/4] selftest: Add new customdc testenv that can load any
 backup-file

This adds a new testenv that can be used for sandpit/manual testing.
This testenv can be based off any backup-file that you like.

The main use case is large databases. Populating 1000s of users is
time-consuming (it can take hours to create a really large DB). Instead
of having to manually add users to the testenv every time you want to
try something, this allows you to populate the users just once, take a
backup/snapshot of the DB, and then spin up the backup multiple times.

In theory this testenv could be useful for other situations too, e.g.
dealing with a corrupted database, testing DB migration (e.g. 4.7 -->
4.8), or if (for some reason) you wanted to create a realistic
lab-domain within a testenv.

To run-up the testenv you need to specify a BACKUP_FILE environment
variable (the same way we specify the SELFTEST_TESTENV), e.g.
  BACKUP_FILE=/files/backup-10k-ad_dc.tar.bz2 \
    SELFTEST_TESTENV=customdc make testenv

Signed-off-by: Tim Beale <timbeale at catalyst.net.nz>
---
 selftest/target/Samba.pm  |   1 +
 selftest/target/Samba4.pm | 100 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 101 insertions(+)

diff --git a/selftest/target/Samba.pm b/selftest/target/Samba.pm
index a3be713..aa6ec9e 100644
--- a/selftest/target/Samba.pm
+++ b/selftest/target/Samba.pm
@@ -423,6 +423,7 @@ sub get_interface($)
     $interfaces{"renamedc"} = 42;
     $interfaces{"labdc"} = 43;
     $interfaces{"offlinebackupdc"} = 44;
+    $interfaces{"customdc"} = 45;
 
     # update lib/socket_wrapper/socket_wrapper.c
     #  #define MAX_WRAPPED_INTERFACES 64
diff --git a/selftest/target/Samba4.pm b/selftest/target/Samba4.pm
index 770ba4f..89704c2 100755
--- a/selftest/target/Samba4.pm
+++ b/selftest/target/Samba4.pm
@@ -12,6 +12,7 @@ use POSIX;
 use SocketWrapper;
 use target::Samba;
 use target::Samba3;
+use Archive::Tar;
 
 sub new($$$$$) {
 	my ($classname, $bindir, $ldap, $srcdir, $server_maxtime) = @_;
@@ -2180,6 +2181,7 @@ sub check_env($$)
 	ad_dc_no_ntlm        => [],
 	ad_dc_ntvfs          => [],
 	backupfromdc         => [],
+	customdc             => [],
 
 	fl2008r2dc           => ["ad_dc"],
 	fl2003dc             => ["ad_dc"],
@@ -3004,6 +3006,104 @@ sub setup_labdc
 	return $env;
 }
 
+# Inspects a backup *.tar.bz2 file and determines the realm/domain it contains
+sub get_backup_domain_realm
+{
+	my ($self, $backup_file) = @_;
+
+	print "Determining REALM/DOMAIN values in backup...\n";
+
+	# The backup will have the correct domain/realm values in the smb.conf.
+	# So we can work out the env variables the testenv should use based on
+	# that. Let's start by extracting the smb.conf
+	my $tar = Archive::Tar->new($backup_file);
+	my $tmpdir = File::Temp->newdir();
+	my $smbconf = "$tmpdir/smb.conf";
+
+	# note that the filepaths within the tar-file differ slightly for online
+	# and offline backups
+	if ($tar->contains_file("etc/smb.conf")) {
+		$tar->extract_file("etc/smb.conf", $smbconf);
+	} elsif ($tar->contains_file("./etc/smb.conf")) {
+		$tar->extract_file("./etc/smb.conf", $smbconf);
+	} else {
+		warn("Could not find smb.conf in $backup_file");
+		return undef, undef;
+	}
+
+	# now use testparm to read the values we're interested in
+	my $testparm = Samba::bindir_path($self, "testparm");
+	my $domain = `$testparm $smbconf -sl --parameter-name=WORKGROUP`;
+	my $realm = `$testparm $smbconf -sl --parameter-name=REALM`;
+	chomp $realm;
+	chomp $domain;
+	print "Backup-file REALM is $realm, DOMAIN is $domain\n";
+
+	return ($domain, $realm);
+}
+
+# This spins up a custom testenv that can be based on any backup-file you want.
+# This is just intended for manual testing (rather than automated test-cases)
+sub setup_customdc
+{
+	my ($self, $prefix) = @_;
+	print "Preparing CUSTOM RESTORE DC...\n";
+	my $dc_name = "customdc";
+	my $password = "locDCpass1";
+	my $backup_file = $ENV{'BACKUP_FILE'};
+
+	# user must specify a backup file to restore via an ENV variable, i.e.
+	# BACKUP_FILE=backup-blah.tar.bz2 SELFTEST_TESTENV=customdc make testenv
+	if (not defined($backup_file)) {
+		warn("Please specify BACKUP_FILE");
+		return undef;
+	}
+
+	# work out the correct domain/realm env values from the backup-file
+	my ($domain, $realm) = $self->get_backup_domain_realm($backup_file);
+
+	# create a placeholder directory and smb.conf, as well as the env vars.
+	my ($env, $ctx) = $self->prepare_dc_testenv($prefix, $dc_name,
+						    $domain, $realm, $password);
+
+	# restore the specified backup file to populate the testenv
+	my $restore_dir = abs_path($prefix);
+	my $ret = $self->restore_backup_file($backup_file,
+					     "--newservername=$env->{SERVER}",
+					     $restore_dir, $env->{SERVERCONFFILE});
+	unless ($ret == 0) {
+		return undef;
+	}
+
+	# Change the admin password to the testenv default, just in case it's
+	# different, or in case this was a --no-secrets backup
+	my $samba_tool = Samba::bindir_path($self, "samba-tool");
+	my $cmd = "$samba_tool user setpassword $env->{USERNAME} ";
+	$cmd .= "--newpassword=$password -H $restore_dir/private/sam.ldb";
+
+	unless(system($cmd) == 0) {
+		warn("Failed to reset admin's password: \n$cmd");
+		return undef;
+	}
+
+	# re-create the testenv's krb5.conf (the restore may have overwritten it,
+	# if the backup-file was an offline backup)
+	Samba::mk_krb5_conf($ctx);
+
+	# start samba for the restored DC
+	if (not defined($self->check_or_start($env, "standard"))) {
+	    return undef;
+	}
+
+	# if this was a backup-rename, then we may need to setup namespaces
+	my $upn_array = ["$env->{REALM}.upn"];
+	my $spn_array = ["$env->{REALM}.spn"];
+
+	$self->setup_namespaces($env, $upn_array, $spn_array);
+
+	return $env;
+}
+
 sub setup_none
 {
 	my ($self, $path) = @_;
-- 
2.7.4


From 7feaa6d31b0a76f7bb15408d02bfbbc8ab328bdf Mon Sep 17 00:00:00 2001
From: Tim Beale <timbeale at catalyst.net.nz>
Date: Tue, 30 Oct 2018 13:06:20 +1300
Subject: [PATCH 2/4] dnsupdate: Pass smb.conf through to samba-tool commands

If you call samba_dnsupdate with a --configfile option, this wasn't
passed through to the samba-tool commands the script tries to run.
Normally, samba_dnsupdate would only be run on the DC itself, so it
shouldn't be a big deal, however, this may be a problem if you install
the samba database into a non-default location (i.e. not
/usr/local/samba).

This patch passes through the smb.conf file, if one was specified.

Signed-off-by: Tim Beale <timbeale at catalyst.net.nz>
---
 source4/scripting/bin/samba_dnsupdate | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/source4/scripting/bin/samba_dnsupdate b/source4/scripting/bin/samba_dnsupdate
index ae355e9..18b6197 100755
--- a/source4/scripting/bin/samba_dnsupdate
+++ b/source4/scripting/bin/samba_dnsupdate
@@ -122,6 +122,7 @@ for i in IPs:
     else:
         IP4s.append(i)
 
+smb_conf = sambaopts.get_loadparm_path()
 
 if opts.verbose:
     print("IPs: %s" % IPs)
@@ -620,6 +621,9 @@ def call_samba_tool(d, op="add", zone=None):
     if d.type == "NS":
         args = [rpc_server_ip, zone, short_name, "NS", d.dest]
 
+    if smb_conf and args:
+        args += ["--configfile=" + smb_conf]
+
     global error_count
     try:
         cmd = cmd_dns()
-- 
2.7.4


From 92aea649cb7c36961700397e271265d1dae00774 Mon Sep 17 00:00:00 2001
From: Tim Beale <timbeale at catalyst.net.nz>
Date: Tue, 30 Oct 2018 14:11:46 +1300
Subject: [PATCH 3/4] dnsupdate: Skip kerberos step if use-file specified

If there's a problem in get_credentials() (getting the machine account
Kerberos credentials), then we fallback to use_samba_tool (essentially
ignoring use-file). However, there's no need to do this, as use-file
shouldn't require Kerberos credentials.

This was making bootstrapping issues starting a testenv harder to debug.
Obviously, Kerberos is dependent on DNS functioning correctly, but
running dnsupdate was also dependent on having a working Kerberos KDC.
In my case, the testenv had a bad krb5.conf file, but the problem
appeared as resolv-wrapper errors (due to a missing RESOLV_WRAPPER_HOSTS
file, which should've been generated by dnsupdate).

Signed-off-by: Tim Beale <timbeale at catalyst.net.nz>
Pair-Programmed-With: Garming Sam <garming at catalyst.net.nz>
---
 source4/scripting/bin/samba_dnsupdate | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/source4/scripting/bin/samba_dnsupdate b/source4/scripting/bin/samba_dnsupdate
index 18b6197..37c8c68 100755
--- a/source4/scripting/bin/samba_dnsupdate
+++ b/source4/scripting/bin/samba_dnsupdate
@@ -196,11 +196,11 @@ def get_credentials(lp):
     creds.set_krb_forwardable(credentials.NO_KRB_FORWARDABLE)
     (tmp_fd, ccachename) = tempfile.mkstemp()
     try:
-        creds.get_named_ccache(lp, ccachename)
-
         if opts.use_file is not None:
             return
 
+        creds.get_named_ccache(lp, ccachename)
+
         # Now confirm we can get a ticket to the DNS server
         get_krb5_rw_dns_server(creds, sub_vars['DNSDOMAIN'] + '.')
         return creds
-- 
2.7.4


From 05755e576f730a36a3e69a565d05d85542272708 Mon Sep 17 00:00:00 2001
From: Tim Beale <timbeale at catalyst.net.nz>
Date: Thu, 1 Nov 2018 16:43:58 +1300
Subject: [PATCH 4/4] selftest: Remove unnecessary code for backup testenvs

setup_namespaces() already gets done for the backupfromdc's domain, so
this step is unnecessary for the restoredc and offlinebackupdc testenvs
(which are based off the backupfromdc's database).

The setup_namespaces() step is still necessary for the renamedc/labdc,
as these don't have the UPN/SPN suffixes for the new realm yet.

Signed-off-by: Tim Beale <timbeale at catalyst.net.nz>
---
 selftest/target/Samba4.pm | 10 ----------
 1 file changed, 10 deletions(-)

diff --git a/selftest/target/Samba4.pm b/selftest/target/Samba4.pm
index 89704c2..41e550b 100755
--- a/selftest/target/Samba4.pm
+++ b/selftest/target/Samba4.pm
@@ -2843,11 +2843,6 @@ sub setup_restoredc
 	    return undef;
 	}
 
-	my $upn_array = ["$env->{REALM}.upn"];
-	my $spn_array = ["$env->{REALM}.spn"];
-
-	$self->setup_namespaces($env, $upn_array, $spn_array);
-
 	return $env;
 }
 
@@ -2939,11 +2934,6 @@ sub setup_offlinebackupdc
 	    return undef;
 	}
 
-	my $upn_array = ["$env->{REALM}.upn"];
-	my $spn_array = ["$env->{REALM}.spn"];
-
-	$self->setup_namespaces($env, $upn_array, $spn_array);
-
 	return $env;
 }
 
-- 
2.7.4



More information about the samba-technical mailing list