[PATCH] script: Add 'random-seed' option to traffic_replay

[Tim Beale]

Hi,

The attach patch adds a new option to the traffic_replay tool that
allows you to seed the random number generator.

When you use a model-file to generate traffic, by default the tool
creates random variation in the packets that it actually generates.
However, this makes it hard to measure changes in samba performance over
time, i.e. differences could be just due to random variation in the
types/number of packets sent, rather than due to the Samba code getting
better/worse. Using the --random-seed option means an identical set of
packets gets produced across several different test runs.

Review appreciated.

Thanks,
Tim

-------------- next part --------------
From c36c9fdd95f834ed9cc342dede6062e4f1821f8e Mon Sep 17 00:00:00 2001
From: Tim Beale <timbeale at catalyst.net.nz>
Date: Wed, 16 May 2018 13:19:58 +1200
Subject: [PATCH] script: Add 'random-seed' option to traffic_replay

When using a traffic-model file to generate traffic, there is some
randomness in the actual packets that get generated. This means it's
hard to use the tool to detect an increase/decrease in Samba
performance - we don't know whether a decrease in packets sent is due
to a regression in the Samba codebase, or just due to the tool sending
different types of packets (i.e. ones that take longer to process).

This patch adds an option to seed the python random number generator.
This means that exactly the same traffic can be generated across
multiple test runs.

(Previously we were using the '--traffic-summary' option to avoid this
problem - we can generate a summary-file based on the model, and then
use the same summary file across multiple runs. However, this proved
impractical when you want to run multiple combinations of scale/rate
parameters, e.g. 21 x 8 different permutations just fills up disk space
with summary-files.)

Signed-off-by: Tim Beale <timbeale at catalyst.net.nz>
---
 docs-xml/manpages/traffic_replay.7.xml | 16 ++++++++++++++--
 script/traffic_replay                  |  6 ++++++
 2 files changed, 20 insertions(+), 2 deletions(-)

diff --git a/docs-xml/manpages/traffic_replay.7.xml b/docs-xml/manpages/traffic_replay.7.xml
index d7d09a2..da8da5c 100644
--- a/docs-xml/manpages/traffic_replay.7.xml
+++ b/docs-xml/manpages/traffic_replay.7.xml
@@ -30,6 +30,7 @@
 		<arg choice="opt">-B, --badpassword-frequency <frequency></arg>
 		<arg choice="opt">--dns-rate <rate></arg>
 		<arg choice="opt">-t, --timing-data <file></arg>
+		<arg choice="opt">--random-seed <seed></arg>
 		<arg choice="opt">-U, --username user</arg>
 		<arg choice="opt">--password <password></arg>
 		<arg choice="opt">-W --workgroup <workgroup></arg>
@@ -137,6 +138,16 @@
 	</varlistentry>
 
 	<varlistentry>
+	<term>random-seed</term>
+	<listitem><para>
+	A number to seed the random number generator with. When traffic is
+	generated from a model-file, use this option to keep the traffic
+	consistent across multiple test runs. This allows you to compare the
+	performance of Samba between different releases.
+	</para></listitem>
+	</varlistentry>
+
+	<varlistentry>
 	<term>Traffic Model Options</term>
 	<listitem><para>
 	When the summary-file is a traffic-model (produced by
@@ -187,7 +198,7 @@
 		<para>
 		For taking performance measurements over several test runs,
 		it's recommended to use this option and replay the traffic from
-		a traffic-summary file.
+		a traffic-summary file, or to use the --random-seed option.
 		</para></listitem>
 		</varlistentry>
 	</itemizedlist>
@@ -474,7 +485,8 @@
 	<para>The traffic generated is an approximation of what was seen in
 	the network capture. The traffic generation involves some randomness,
 	so running the same command multiple times may result in slightly
-	different traffic being generated.</para>
+	different traffic being generated (although you can avoid this, by
+	specifying the --random-seed option).</para>
 
 	<para>As well as changing how long the model runs for, you can also
 	change how many conversations get generated and how fast the traffic
diff --git a/script/traffic_replay b/script/traffic_replay
index 6f42f2d..764462e 100755
--- a/script/traffic_replay
+++ b/script/traffic_replay
@@ -22,6 +22,7 @@ import os
 import optparse
 import tempfile
 import shutil
+import random
 
 sys.path.insert(0, "bin/python")
 
@@ -67,6 +68,8 @@ def main():
     parser.add_option('-c', '--clean-up',
                       action="store_true",
                       help='Clean up the generated groups and user accounts')
+    parser.add_option('--random-seed', type='int', default=0,
+                      help='Use to keep randomness consistent across multiple runs')
 
     model_group = optparse.OptionGroup(parser, 'Traffic Model Options',
                                        'These options alter the traffic '
@@ -155,6 +158,9 @@ def main():
                              " for the users created as part of this test"))
         sys.exit(1)
 
+    if opts.random_seed:
+        random.seed(opts.random_seed)
+
     lp = sambaopts.get_loadparm()
     creds = credopts.get_credentials(lp)
     creds.set_gensec_features(creds.get_gensec_features() | gensec.FEATURE_SEAL)
-- 
2.7.4