Performance testing framework

[Douglas Bagnall]

I have been mangling selftest to work in a mode that outputs
performance timings and nothing else, and I have managed to backport
this as far as 4.0 (which seems a long time ago to *me*). I've made a
set of tests that exercise some simple AD stuff, and without further
ado, here are some graphs:

https://www.samba.org/~dbagnall/perf-tests/

If for any reason you did not see, the graphs show that 4.0 to 4.3 are
mostly similar, 4.4 is slow, and 4.5 is fast. Many operations
involving many objects with linked attributes are roughly twice as
fast in 4.5. This of course represents tester's bias: we've been
working on that and want to show it.

The tests are plain subunit tests, so any existing test could be
tracked (add it to selftest/perf_tests.py), but beware of tests that
change over time or that came about with actual fixes. I chose to test
very simple things that should work in every version.

The times are formatted as JSON. I can hear you falling off your
chairs. JSON?! Don't we have a perfectly good serialization format in
ASN.1? Well, the aim is to generate automatic performance charts, akin
to http://speed.pypy.org/ or https://arewefastyet.com/ or
http://www.astropy.org/astropy-benchmarks/. Javascript graphing
libraries really like JSON, and it is actually quite simple to deal
with in Python. I would like to run the tests more often -- possibly
on every autobuild commit, or at least daily or weekly. We have people
using Samba in large deployments, and we would prefer to discover
scalability regressions before they grind some vast organisation to a
halt.

The scripts I used to run the tests and make the graphs are not in the
attached patch, and not in a Samba tree. Instead they're in a messy
tools repository we keep at Catalyst. (e.g.:
http://git.catalyst.net.nz/gitweb?p=samba-cloud-autobuild.git;a=blob;f=scripts/multi-perf-test;h=64f6d2d6e6;hb=refs/heads/perf).
They could go in the Samba repository but it gets a bit meta -- the
script spawns and patches copies of the Samba repository, which could
get confusing if run inside a Samba repo.

And please, if you want to see how print spool performance has fared
over time, add your test to selftest/perf_tests.py. The tests need to
be reasonably slow (in the order of seconds) to avoid selftest
overhead noise, and they need to be unchanging. It doesn't matter if
they're old and unchanging or new and unchanging and cleanly apply in
old trees, so long as they do the same thing in all the tested
versions.

Douglas
-------------- next part --------------
A non-text attachment was scrubbed...
Name: perf-testing.patch
Type: text/x-diff
Size: 25279 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20160819/c92c2186/perf-testing-0001.diff>