[SCM] Samba Shared Repository - branch master updated
Douglas Bagnall
dbagnall at samba.org
Wed Jan 9 02:54:03 UTC 2019
The branch, master has been updated
via b2a9d4c1f69 xml_docs: update traffic script documentation
via 3c10cecac10 traffic_replay: use packets per second as primary scale
via c672a922618 traffic: assert json equality in learner test
via daf039afcb1 traffic tests: use TestCaseInTempdir.mktemp()
via 71cb2605e86 python/tests: add TestCaseInTempdir.mktemp()
via affaeb944fd traffic_replay: --old-scale to mimic the old traffic_replay
via 524777e681b traffic: add option to reanimate dying conversations
via bda7f35a5e1 traffic_model: don't report generation errors as parse errors
via 9e0effc1757 traffic: improve debug messages in traffic_replay
via b5d493f9272 traffic: Conversation.add_short_packet is discerning about packets
via db51004307a traffic: remove useless branch in stats report
via d727dcc82a8 traffic: generate more statistics in replay
via 31c0809e4f1 traffic: avoid bare except: clauses
via 41b0cfce2c6 traffic: simplify tests for badpassword_frequency
via 65c02fd68c3 traffic: assign context domain at start
via 17579dd8105 traffic: traffic_replay --latency-timeout to control final wait
via b737552ed3a traffic_replay: --stop-on-any-error option to not ignore client trouble
via 22cba7011ca traffic: evoke OS error codes by name, not number
via 35bc6ee0a10 traffic: reinitialise RNG in each child
via 6268effb110 traffic: avoid generating conversations without packets
via c4d5bb59527 traffic: rework conversation generation to better use memory
via 7b03e81c615 traffic: generate sparser descriptions of conversations
via 7edf58dc586 traffic: new version of model with packet_rate, version number
via 273eb3dffba traffic: fix hash non-determinism when loading JSON
via ef747953d4b traffic: Conversation gets given id, list of short packets
via a430b11ca2e traffic: rename packet_rate -> replay_speed for accuracy and room
via a352060f974 traffic: initialise conversation specific lists after the fork
via e866782a153 traffic: split is_a_real_packet() function out of class
via 24d09d1df84 traffic: give Packet __slots__ for reduced memory
via 4f7ae5d9256 traffic: make random_colour_print more deterministic, silent on demand
via b77ee628cc8 traffic: delete empty temp directories
via 158f172845f traffic: use namedtuple for ConversationAccounts namespace class
via 9da79b54fa4 traffic: fix mk_masked_dir doc and comments
via 709d2c84cb3 traffic: Packet.from_line classmethod uses cls var
via d1a1c5d6011 traffic learner: avoid truncated output files on error
via b275baebd7e traffic_learner: use samba.logger, not print(file=sys.stderr)
via 3b3b4fa6f96 traffic_learner: return an error code
via f6b91d5b21b traffic: use default value for --duration
via 79e84598172 traffic_replay: drop summary replay
via 4c463334997 traffic replay: allow --random-seed=0
via 8760168aa84 python/traffic: remove duplicate test file
via c57f3794e24 python/tests/blackbox: check_exit_code() explains by default
from ee4a33a4fff idmap: In _wbint_Sids2UnixIDs, pass back what we have
https://git.samba.org/?p=samba.git;a=shortlog;h=master
- Log -----------------------------------------------------------------
commit b2a9d4c1f69ef5453cec5771c2a68d50ab83434f
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Fri Jan 4 15:14:58 2019 +1300
xml_docs: update traffic script documentation
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Tim Beale <timbeale at samba.org>
Autobuild-User(master): Douglas Bagnall <dbagnall at samba.org>
Autobuild-Date(master): Wed Jan 9 03:53:58 CET 2019 on sn-devel-144
commit 3c10cecac1005d1340f1a389284c30b88cd73910
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Wed Nov 7 13:31:31 2018 +1300
traffic_replay: use packets per second as primary scale
The old -S/--scale-traffic is relative to the original model, which made
its relationship to true traffic volumes quite opaque
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit c672a922618507d41e2630c3b3c1407b274b153d
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Mon Dec 3 13:58:56 2018 +1300
traffic: assert json equality in learner test
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit daf039afcb1ba21ce49eaaff9589c5e7c667279d
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Thu Oct 25 13:37:02 2018 +1300
traffic tests: use TestCaseInTempdir.mktemp()
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 71cb2605e86d04e489fbb8081daff8f7b8e56ae1
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Thu Oct 25 13:33:02 2018 +1300
python/tests: add TestCaseInTempdir.mktemp()
This gives you a name of a temporary file within the test case's tempdir.
Use it like this:
with self.mktemp() as filename:
self.check_run("samba-tool foo --output %s" % filename)
self.assertStringsEqual(open(filename).read(), expected)
and filename will flick out of existence when the with block ends.
This is based on an idea used in the traffic_runner tests, which will
soon be adapted to use this method.
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit affaeb944fd77ea67d6fb9ece2c53e6433e3b4a7
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Tue Oct 23 10:30:46 2018 +1300
traffic_replay: --old-scale to mimic the old traffic_replay
traffic_replay had a broken sense of traffic scale. That is fixed, but
in order to compare old and new tests, it helps to be able to
approximate the old behaviour.
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 524777e681b1fecf5686843f10ae01637285a891
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Tue Oct 23 11:58:52 2018 +1300
traffic: add option to reanimate dying conversations
The traffic model is generated from a window in time, which makes
conversations appear to start and stop unnaturally at the window
boundaries. When the window is short compared to the traffic replay
time and the true expected conversation length, this has a significant
distorting effect, leading to more conversations than would be
expected to generate a given number of packets.
To offset this slightly we add the --conversation-persistence option
which tries to convert apparent death into a longish wait.
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit bda7f35a5e1418a3e9d4f582f72d2bbe9cb10f44
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Tue Oct 23 11:00:20 2018 +1300
traffic_model: don't report generation errors as parse errors
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 9e0effc1757cd1c5f839a90eb04c4a481d518207
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Fri Oct 19 16:26:19 2018 +1300
traffic: improve debug messages in traffic_replay
tracebacks and less nonsense at higher debug levels.
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit b5d493f92723c1c52616d2497953e4b7da0932ec
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Fri Oct 19 17:24:53 2018 +1300
traffic: Conversation.add_short_packet is discerning about packets
If the packets really wouldn't do anything, we might as well not add them.
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit db51004307a19447bb0ca2259cd7a49f4efc1874
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Thu Nov 22 15:09:00 2018 +1300
traffic: remove useless branch in stats report
This completes the work of 68c64c634ad7576b2ab5ef838c203320afd90f44,
but differs from that in that it makes no actual change because isatty
was not being called so was always evaluated as true.
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit d727dcc82a8843d72922ae3f89b6a778dff9d565
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Fri Oct 19 17:21:56 2018 +1300
traffic: generate more statistics in replay
Add more "header" values indicating the progress of the run as a
whole.
The new fields are:
Max sleep miss - the longest sleep() oversleep. Indicates client load.
Maximum lag - the longest gap between a planned packet
time and its actual time.
Start lag - the longest gap between intended and actual
conversation start.
Planned conversations - how many conversations we meant to have.
Planned packets - how many "packets" we thought we were making. Not
all "packets" result in actual operations or packets.
Unfinished conversations - how many conversations had not finished
when they were killed.
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 31c0809e4f1760f731006be6aca8c6da34edab24
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Fri Oct 19 17:19:29 2018 +1300
traffic: avoid bare except: clauses
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 41b0cfce2c63ebc8f0c061448f58499b47e14485
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Fri Oct 19 17:19:09 2018 +1300
traffic: simplify tests for badpassword_frequency
x <= 0 will fail one or both of the other test clauses.
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 65c02fd68c337779b2482f3bfc7138372eac2d70
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Fri Oct 19 17:15:30 2018 +1300
traffic: assign context domain at start
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 17579dd81052813cde2ed3aad0738540d417e60b
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Tue Oct 23 10:31:53 2018 +1300
traffic: traffic_replay --latency-timeout to control final wait
Conversations that haven't finished within some acceptable margin of
on-time can be said to have failed. This is where you specify that
margin.
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit b737552ed3a60c66d44e0b6d93da66a52682db39
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Fri Oct 19 16:27:17 2018 +1300
traffic_replay: --stop-on-any-error option to not ignore client trouble
Sometimes you want to know if any client is crashing for any reason.
In those times use --stop-on-any-error for an early exit.
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 22cba7011caa50b32f35d221ea1020e18f506c1d
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Thu Nov 22 13:00:04 2018 +1300
traffic: evoke OS error codes by name, not number
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 35bc6ee0a105845a74d0475ed5d3f5aca81bf7d1
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Thu Nov 22 12:19:46 2018 +1300
traffic: reinitialise RNG in each child
Otherwise they all replay using the same random sequence.
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 6268effb110598375f4f04b8b8758143532e35cb
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Wed Oct 24 14:02:01 2018 +1300
traffic: avoid generating conversations without packets
Some "packets" don't generate any actual traffic. If we have a
conversation consisting only of those, we can avoid forking a client
for it.
This *slightly* increases the load over that which would be generated
otherwise for a given traffic rate, but that's OK.
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit c4d5bb59527366061a795e99667091018bd6cc99
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Fri Oct 19 17:11:52 2018 +1300
traffic: rework conversation generation to better use memory
Use less memory altogether and don't allocated shared mutable before
the fork.
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 7b03e81c615a9b3ae24f90019c9b269b498bafc0
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Fri Dec 14 14:26:34 2018 +1300
traffic: generate sparser descriptions of conversations
Rather than building all the packets at this point, we stick to the
barest details of the packets (which is all the model gives us
anyway).
The advantage is that will take a lot less memory, which matters
because this process forks into many clients that were sharing and
mutate the conversation list.
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 7edf58dc58630eca4f9b11242035aa5d6d24e285
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Thu Dec 20 17:25:49 2018 +1300
traffic: new version of model with packet_rate, version number
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 273eb3dffbaff02b9cc4246783153e26d68485c5
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Thu Jan 3 15:54:37 2019 +1300
traffic: fix hash non-determinism when loading JSON
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Tim Beale <timbeale at samba.org>
commit ef747953d4b30bcfa691a52170caae1062b68502
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Fri Oct 19 17:00:06 2018 +1300
traffic: Conversation gets given id, list of short packets
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit a430b11ca2e450792ae9617453f4971ce50e36c1
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Mon Dec 3 13:32:59 2018 +1300
traffic: rename packet_rate -> replay_speed for accuracy and room
We are soon going to have a self.packet_rate, and replay_speed is more
accurate in this case.
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit a352060f9748cd8f26dbd48f57eff9edf62e52b4
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Fri Oct 19 16:55:59 2018 +1300
traffic: initialise conversation specific lists after the fork
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit e866782a15360784f495ac2e1451aa1b98d94f8b
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Fri Oct 19 16:52:48 2018 +1300
traffic: split is_a_real_packet() function out of class
So we can use it to determine whether a packet should be a Packet before
making the leap.
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 24d09d1df8474cd3ef011bbc4cd342ac2561198b
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Fri Oct 19 16:51:22 2018 +1300
traffic: give Packet __slots__ for reduced memory
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 4f7ae5d925650149769aa42fedba56b9c3ee8d3e
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Fri Oct 19 16:48:40 2018 +1300
traffic: make random_colour_print more deterministic, silent on demand
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit b77ee628cc893bea64f72e7035d21904909e8cc1
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Fri Oct 19 16:25:06 2018 +1300
traffic: delete empty temp directories
even if someone asked to keep the temporary data, they don't want
to see 5000 empty directries. Non-empty directories will remain.
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 158f172845f6b2cbe0b392cc23fde9bdd32960d4
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Fri Oct 19 16:46:05 2018 +1300
traffic: use namedtuple for ConversationAccounts namespace class
Saving memory, which reduces fork overhead.
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 9da79b54fa4ec31d82b7d1a23568b576140a7e9b
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Fri Oct 19 16:43:13 2018 +1300
traffic: fix mk_masked_dir doc and comments
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 709d2c84cb3b3c7a570b0f915b578bb98e48e3f0
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Fri Oct 19 16:41:36 2018 +1300
traffic: Packet.from_line classmethod uses cls var
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit d1a1c5d60113727fad1208c91060f4d318dea412
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Fri Oct 19 16:39:36 2018 +1300
traffic learner: avoid truncated output files on error
add_argument(type=argparse.FileType('w'), ...) will open the file
and leave it empty if the script fails.
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit b275baebd7e7fa8576cb49737a886f02e3615640
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Fri Oct 19 16:37:08 2018 +1300
traffic_learner: use samba.logger, not print(file=sys.stderr)
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 3b3b4fa6f960d2bc06d830266671b582ddf186a1
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Wed Oct 24 13:46:16 2018 +1300
traffic_learner: return an error code
And use it in tests, rather than expecting exact strings.
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit f6b91d5b21b660217e792bf75b6fcb435232c383
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Fri Oct 19 16:03:25 2018 +1300
traffic: use default value for --duration
We could not do this when we replayed traffic summaries (as opposed to
models), but now this script does not do that.
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 79e8459817246a80918dca73b109549475e128fc
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Fri Oct 19 16:18:39 2018 +1300
traffic_replay: drop summary replay
The traffic_replay script has been able to replay a replay log as well
as a model, which was not used in practice and complicated the script.
If we want that feature, we can make a new script for it.
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 4c463334997d184f6186d26258bdde84d216e136
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Fri Oct 5 14:42:27 2018 +1300
traffic replay: allow --random-seed=0
Zero is a valid seed.
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit 8760168aa842c1e883e2059036c7e6621c77d887
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Thu Sep 27 12:58:11 2018 +1200
python/traffic: remove duplicate test file
The other one is python/samba/tests/blackbox/testdata/traffic-sample-very-short.model
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
commit c57f3794e240704621c67418e018fee2775d1844
Author: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Date: Wed Oct 24 17:49:00 2018 +1300
python/tests/blackbox: check_exit_code() explains by default
Signed-off-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet at samba.org>
-----------------------------------------------------------------------
Summary of changes:
docs-xml/manpages/traffic_learner.7.xml | 3 +-
docs-xml/manpages/traffic_replay.7.xml | 83 ++-
python/samba/emulate/traffic.py | 820 +++++++++++++--------
python/samba/tests/__init__.py | 19 +-
.../testdata/traffic-sample-very-short.model | 11 +-
.../blackbox/testdata/traffic_learner.expected | 45 +-
.../blackbox/testdata/traffic_replay-0.expected | 18 +
.../blackbox/testdata/traffic_replay-1.expected | 19 +
.../blackbox/testdata/traffic_replay-2.expected | 17 +
.../blackbox/testdata/traffic_replay-3.expected | 11 +
.../blackbox/testdata/traffic_replay.expected | 28 +-
python/samba/tests/blackbox/traffic_learner.py | 34 +-
python/samba/tests/blackbox/traffic_replay.py | 78 +-
python/samba/tests/blackbox/traffic_summary.py | 17 +-
script/traffic_learner | 28 +-
script/traffic_replay | 153 ++--
testdata/traffic-sample-very-short.model | 60 --
17 files changed, 842 insertions(+), 602 deletions(-)
create mode 100644 python/samba/tests/blackbox/testdata/traffic_replay-0.expected
create mode 100644 python/samba/tests/blackbox/testdata/traffic_replay-1.expected
create mode 100644 python/samba/tests/blackbox/testdata/traffic_replay-2.expected
create mode 100644 python/samba/tests/blackbox/testdata/traffic_replay-3.expected
delete mode 100644 testdata/traffic-sample-very-short.model
Changeset truncated at 500 lines:
diff --git a/docs-xml/manpages/traffic_learner.7.xml b/docs-xml/manpages/traffic_learner.7.xml
index 6fd7975393d..b921035e4bd 100644
--- a/docs-xml/manpages/traffic_learner.7.xml
+++ b/docs-xml/manpages/traffic_learner.7.xml
@@ -118,8 +118,7 @@
</para>
<para>The model is stored in JSON format, and also contains
- information about the conversation rate and DNS traffic
- rate.</para>
+ information about the packet rate and DNS traffic rate.</para>
<refsect2>
diff --git a/docs-xml/manpages/traffic_replay.7.xml b/docs-xml/manpages/traffic_replay.7.xml
index da8da5cf3c6..167bd9e0b08 100644
--- a/docs-xml/manpages/traffic_replay.7.xml
+++ b/docs-xml/manpages/traffic_replay.7.xml
@@ -21,6 +21,7 @@
<cmdsynopsis>
<command>traffic_replay</command>
<arg choice="opt">-F, --fixed-password <test-password></arg>
+ <arg choice="opt">-T, --packets-per-second <number></arg>
<arg choice="opt">-S, --scale-traffic <scale by factor></arg>
<arg choice="opt">-r, --replay-rate <scale by factor></arg>
<arg choice="opt">-D, --duration <seconds></arg>
@@ -41,6 +42,9 @@
<arg choice="opt">-P, --machine-pass</arg>
<arg choice="opt">--option <option></arg>
<arg choice="opt">-d, --debuglevel <debug level></arg>
+ <arg choice="opt">--conversation-persistence <0-1></arg>
+ <arg choice="opt">--latency-timeout <seconds></arg>
+ <arg choice="opt">--stop-on-any-error</arg>
<arg choice="req">summary-file</arg>
<arg choice="req">dns-hostname</arg>
</cmdsynopsis>
@@ -53,6 +57,7 @@
<arg choice="opt">--number-of-groups <total groups></arg>
<arg choice="opt">--average-groups-per-user <average number></arg>
<arg choice="opt">--group-memberships <total memberships></arg>
+ <arg choice="opt">--max-members <group size></arg>
<arg choice="req">dns-hostname</arg>
</cmdsynopsis>
@@ -104,11 +109,10 @@
<varlistentry>
<term>summary-file</term>
<listitem><para>
- File containing the network traffic to replay. This should either be
- a traffic-summary (generated by <command>traffic_summary.pl</command>)
- or a traffic-model (generated by <command>traffic_learner</command>).
- Based on this file, this tool will generate 'conversations' which
- represent Samba activity between a network host and the DC.
+ File containing the network traffic to replay. This should be a
+ traffic-model (generated by <command>traffic_learner</command>).
+ Based on this file, this tool will generate 'conversations' which
+ represent Samba activity between a network host and the DC.
</para></listitem>
</varlistentry>
@@ -153,7 +157,7 @@
When the summary-file is a traffic-model (produced by
<command>traffic_learner</command>), use these options to alter the
traffic that gets generated.</para>
- <itemizedlist>
+ <variablelist>
<varlistentry>
<term>-D|--duration <seconds></term>
<listitem><para>
@@ -163,22 +167,34 @@
</varlistentry>
<varlistentry>
- <term>-r|--replay-rate <factor></term>
+ <term>-T|--packets-per-second <number></term>
<listitem><para>
- Replays the traffic faster by this factor. This option won't
- affect the number of conversations (which is based on the
- traffic model), but the rate at which the packets are sent will
- be increased.
+ Generate this many packets per second, regardless of
+ the traffic rate of the sample on which the model
+ was based. This cannot be used with <option>-S</option>.
</para></listitem>
</varlistentry>
<varlistentry>
<term>-S|--scale-traffic <factor></term>
<listitem><para>
- Increases the number of conversations by this factor. This
- option won't affect the rate at which packets get sent (which
- is still based on the traffic model), but it will mean more
- conversations get replayed.
+ Increases the number of conversations by this factor,
+ relative to the original traffic sample on which the
+ model was based. This option won't affect the rate at
+ which packets get sent (which is still based on the
+ traffic model), but it will mean more conversations
+ get replayed. It cannot be combined with
+ <option>-T</option>, which sets the traffic rate in a
+ different way.
+ </para></listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>-r|--replay-rate <factor></term>
+ <listitem><para> Replays the traffic faster by this
+ factor. This option won't affect the number of packets
+ sent, but it will squeeze them into fewer
+ conversations, which may reduce resource usage.
</para></listitem>
</varlistentry>
@@ -201,7 +217,28 @@
a traffic-summary file, or to use the --random-seed option.
</para></listitem>
</varlistentry>
- </itemizedlist>
+ <varlistentry>
+ <term>--stop-on-any-error</term>
+ <listitem><para> Any client error causes the whole run to stop.
+ </para></listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>--conversation-persistence <0-1></term>
+ <listitem><para> Conversation termination (as decided
+ by the model) is re-interpreted as a long pause with
+ this probability. </para></listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>--latency-timeout <seconds></term>
+ <listitem><para> Wait this long at the end of the run
+ for outstanding reply packets. The number of
+ conversations that have not finished at the end of the
+ timeout is a failure metric. </para></listitem>
+ </varlistentry>
+
+ </variablelist>
</listitem>
</varlistentry>
@@ -214,7 +251,7 @@
not actually used for traffic generation - the traffic generation is
still based on the number of conversations from the model/summary file.
</para>
-
+
<para>
Generating a large number of users can take a long time, so it this
option allows this to be done only once.</para>
@@ -241,7 +278,7 @@
Creates the specified number of groups, for assigning the test
users to. Note that users are not automatically assigned to
groups - use either --average-groups-per-user or
- --group-membership to do this.
+ --group-memberships to do this.
</para></listitem>
</varlistentry>
@@ -268,7 +305,13 @@
on average. Some users will belong to more groups and some
users will belong to fewer groups, but the total of all
member linked attributes would be 300. This option is
- incompatible with the --group-membership option.
+ incompatible with the --average-groups-per-user option.
+ </para></listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>--max-members <group size></term>
+ <listitem><para> Limit the largest group to this size,
+ even if the other group options would have it otherwise.
</para></listitem>
</varlistentry>
</itemizedlist>
@@ -422,7 +465,7 @@
along with the full DNS hostname of the DC being tested. You also need
to provide some user credentials, and possibly the Samba realm and
workgroup (although the realm and workgroup may be determined
- automatically, for example from the /etc/smb.conf file, if one is
+ automatically, for example from the /etc/smb.conf file, if one is
present). E.g.</para>
<para><command>traffic_replay traffic-summary.txt
diff --git a/python/samba/emulate/traffic.py b/python/samba/emulate/traffic.py
index 291162f279a..03603759816 100644
--- a/python/samba/emulate/traffic.py
+++ b/python/samba/emulate/traffic.py
@@ -25,9 +25,9 @@ import json
import math
import sys
import signal
-import itertools
+from errno import ECHILD, ESRCH
-from collections import OrderedDict, Counter, defaultdict
+from collections import OrderedDict, Counter, defaultdict, namedtuple
from samba.emulate import traffic_packets
from samba.samdb import SamDB
import ldb
@@ -55,6 +55,8 @@ from samba.compat import get_string
from samba.logger import get_samba_logger
import bisect
+CURRENT_MODEL_VERSION = 2 # save as this
+REQUIRED_MODEL_VERSION = 2 # load accepts this or greater
SLEEP_OVERHEAD = 3e-4
# we don't use None, because it complicates [de]serialisation
@@ -126,14 +128,26 @@ def debug_lineno(*args):
sys.stderr.flush()
-def random_colour_print():
- """Return a function that prints a randomly coloured line to stderr"""
- n = 18 + random.randrange(214)
- prefix = "\033[38;5;%dm" % n
-
- def p(*args):
- for a in args:
- print("%s%s\033[00m" % (prefix, a), file=sys.stderr)
+def random_colour_print(seeds):
+ """Return a function that prints a coloured line to stderr. The colour
+ of the line depends on a sort of hash of the integer arguments."""
+ if seeds:
+ s = 214
+ for x in seeds:
+ s += 17
+ s *= x
+ s %= 214
+ prefix = "\033[38;5;%dm" % (18 + s)
+
+ def p(*args):
+ if DEBUG_LEVEL > 0:
+ for a in args:
+ print("%s%s\033[00m" % (prefix, a), file=sys.stderr)
+ else:
+ def p(*args):
+ if DEBUG_LEVEL > 0:
+ for a in args:
+ print(a, file=sys.stderr)
return p
@@ -144,9 +158,18 @@ class FakePacketError(Exception):
class Packet(object):
"""Details of a network packet"""
+ __slots__ = ('timestamp',
+ 'ip_protocol',
+ 'stream_number',
+ 'src',
+ 'dest',
+ 'protocol',
+ 'opcode',
+ 'desc',
+ 'extra',
+ 'endpoints')
def __init__(self, timestamp, ip_protocol, stream_number, src, dest,
protocol, opcode, desc, extra):
-
self.timestamp = timestamp
self.ip_protocol = ip_protocol
self.stream_number = stream_number
@@ -162,7 +185,7 @@ class Packet(object):
self.endpoints = (self.dest, self.src)
@classmethod
- def from_line(self, line):
+ def from_line(cls, line):
fields = line.rstrip('\n').split('\t')
(timestamp,
ip_protocol,
@@ -178,8 +201,8 @@ class Packet(object):
src = int(src)
dest = int(dest)
- return Packet(timestamp, ip_protocol, stream_number, src, dest,
- protocol, opcode, desc, extra)
+ return cls(timestamp, ip_protocol, stream_number, src, dest,
+ protocol, opcode, desc, extra)
def as_summary(self, time_offset=0.0):
"""Format the packet as a traffic_summary line.
@@ -276,32 +299,56 @@ class Packet(object):
return self.timestamp - other.timestamp
def is_really_a_packet(self, missing_packet_stats=None):
- """Is the packet one that can be ignored?
+ return is_a_real_packet(self.protocol, self.opcode)
- If so removing it will have no effect on the replay
- """
- if self.protocol in SKIPPED_PROTOCOLS:
- # Ignore any packets for the protocols we're not interested in.
- return False
- if self.protocol == "ldap" and self.opcode == '':
- # skip ldap continuation packets
- return False
- fn_name = 'packet_%s_%s' % (self.protocol, self.opcode)
- fn = getattr(traffic_packets, fn_name, None)
- if not fn:
- print("missing packet %s" % fn_name, file=sys.stderr)
- return False
- if fn is traffic_packets.null_packet:
- return False
- return True
+def is_a_real_packet(protocol, opcode):
+ """Is the packet one that can be ignored?
+
+ If so removing it will have no effect on the replay
+ """
+ if protocol in SKIPPED_PROTOCOLS:
+ # Ignore any packets for the protocols we're not interested in.
+ return False
+ if protocol == "ldap" and opcode == '':
+ # skip ldap continuation packets
+ return False
+
+ fn_name = 'packet_%s_%s' % (protocol, opcode)
+ fn = getattr(traffic_packets, fn_name, None)
+ if fn is None:
+ LOGGER.debug("missing packet %s" % fn_name, file=sys.stderr)
+ return False
+ if fn is traffic_packets.null_packet:
+ return False
+ return True
+
+
+def is_a_traffic_generating_packet(protocol, opcode):
+ """Return true if a packet generates traffic in its own right. Some of
+ these will generate traffic in certain contexts (e.g. ldap unbind
+ after a bind) but not if the conversation consists only of these packets.
+ """
+ if protocol == 'wait':
+ return False
+
+ if (protocol, opcode) in (
+ ('kerberos', ''),
+ ('ldap', '2'),
+ ('dcerpc', '15'),
+ ('dcerpc', '16')):
+ return False
+
+ return is_a_real_packet(protocol, opcode)
class ReplayContext(object):
- """State/Context for an individual conversation between an simulated client
- and a server.
+ """State/Context for a conversation between an simulated client and a
+ server. Some of the context is shared amongst all conversations
+ and should be generated before the fork, while other context is
+ specific to a particular conversation and should be generated
+ *after* the fork, in generate_process_local_config().
"""
-
def __init__(self,
server=None,
lp=None,
@@ -312,17 +359,9 @@ class ReplayContext(object):
statsdir=None,
ou=None,
base_dn=None,
- domain=None,
+ domain=os.environ.get("DOMAIN"),
domain_sid=None):
-
self.server = server
- self.ldap_connections = []
- self.dcerpc_connections = []
- self.lsarpc_connections = []
- self.lsarpc_connections_named = []
- self.drsuapi_connections = []
- self.srvsvc_connections = []
- self.samr_contexts = []
self.netlogon_connection = None
self.creds = creds
self.lp = lp
@@ -346,7 +385,6 @@ class ReplayContext(object):
self.last_netlogon_bad = False
self.last_samlogon_bad = False
self.generate_ldap_search_tables()
- self.next_conversation_id = itertools.count()
def generate_ldap_search_tables(self):
session = system_session()
@@ -399,8 +437,13 @@ class ReplayContext(object):
self.attribute_clue_map = attribute_clue_map
def generate_process_local_config(self, account, conversation):
- if account is None:
- return
+ self.ldap_connections = []
+ self.dcerpc_connections = []
+ self.lsarpc_connections = []
+ self.lsarpc_connections_named = []
+ self.drsuapi_connections = []
+ self.srvsvc_connections = []
+ self.samr_contexts = []
self.netbios_name = account.netbios_name
self.machinepass = account.machinepass
self.username = account.username
@@ -415,11 +458,6 @@ class ReplayContext(object):
self.lp.set("state directory", self.tempdir)
self.lp.set("tls verify peer", "no_check")
- # If the domain was not specified, check for the environment
- # variable.
- if self.domain is None:
- self.domain = os.environ["DOMAIN"]
-
self.remoteAddress = "/root/ncalrpc_as_system"
self.samlogon_dn = ("cn=%s,%s" %
(self.netbios_name, self.ou))
@@ -442,11 +480,11 @@ class ReplayContext(object):
than that requested, but not significantly.
"""
if not failed_last_time:
- if (self.badpassword_frequency and self.badpassword_frequency > 0
- and random.random() < self.badpassword_frequency):
+ if (self.badpassword_frequency and
+ random.random() < self.badpassword_frequency):
try:
f(bad)
- except:
+ except Exception:
# Ignore any exceptions as the operation may fail
# as it's being performed with bad credentials
pass
@@ -724,12 +762,23 @@ class ReplayContext(object):
def get_authenticator(self):
auth = self.machine_creds.new_client_authenticator()
current = netr_Authenticator()
- current.cred.data = [x if isinstance(x, int) else ord(x) for x in auth["credential"]]
+ current.cred.data = [x if isinstance(x, int) else ord(x)
+ for x in auth["credential"]]
current.timestamp = auth["timestamp"]
subsequent = netr_Authenticator()
return (current, subsequent)
+ def write_stats(self, filename, **kwargs):
+ """Write arbitrary key/value pairs to a file in our stats directory in
+ order for them to be picked up later by another process working out
+ statistics."""
+ filename = os.path.join(self.statsdir, filename)
+ f = open(filename, 'w')
+ for k, v in kwargs.items():
+ print("%s: %s" % (k, v), file=f)
+ f.close()
+
class SamrContext(object):
"""State/Context associated with a samr connection.
@@ -764,14 +813,16 @@ class SamrContext(object):
class Conversation(object):
"""Details of a converation between a simulated client and a server."""
- conversation_id = None
-
- def __init__(self, start_time=None, endpoints=None):
+ def __init__(self, start_time=None, endpoints=None, seq=(),
+ conversation_id=None):
self.start_time = start_time
self.endpoints = endpoints
self.packets = []
- self.msg = random_colour_print()
+ self.msg = random_colour_print(endpoints)
self.client_balance = 0.0
+ self.conversation_id = conversation_id
+ for p in seq:
+ self.add_short_packet(*p)
def __cmp__(self, other):
if self.start_time is None:
@@ -809,11 +860,14 @@ class Conversation(object):
self.packets.append(p)
def add_short_packet(self, timestamp, protocol, opcode, extra,
- client=True):
+ client=True, skip_unused_packets=True):
"""Create a packet from a timestamp, and 'protocol:opcode' pair, and a
(possibly empty) list of extra data. If client is True, assume
this packet is from the client to the server.
"""
+ if skip_unused_packets and not is_a_real_packet(protocol, opcode):
+ return
+
src, dest = self.guess_client_server()
if not client:
src, dest = dest, src
@@ -860,86 +914,41 @@ class Conversation(object):
lines.append(p.as_summary(self.start_time))
return lines
- def replay_in_fork_with_delay(self, start, context=None, account=None):
- """Fork a new process and replay the conversation.
--
Samba Shared Repository
More information about the samba-cvs
mailing list