Improving the speed of make test

[Amitay Isaacs]

On Tue, Mar 6, 2012 at 10:05 AM, Jelmer Vernooij <jelmer at samba.org> wrote:
> Am 05/03/12 21:44, schrieb Andrew Bartlett:
>
>> On Mon, 2012-03-05 at 17:44 +0100, Jelmer Vernooij wrote:
>>>
>>> On 03/05/2012 08:36 AM, Andrew Bartlett wrote:
>>>>
>>>> Jelmer and others interested in selftest:
>>>>
>>>> A while back, I started doing some profiling to determine where the time
>>>> is spent in 'make test', using 'perf'.
>>>>
>>>> I was surprised to find that 15% of our time is spent in routines
>>>> associated with SHA1, due to adding users and kinit.  Both of these run
>>>> a *lot* of SHA1, because salting the password for the AES-based kerberos
>>>> keys uses multiple thousands of rounds of SHA1, to make brute forcing
>>>> the password hash harder.
>>>>
>>>> The fix is simple:
>>>>   - change acl.py and similar tests not to create a user for each unit
>>>> test, but re-use one for the whole testsuite
>>>>   - kinit once at the start of make test, for all connections that
>>>> should
>>>> be made as administrator.  Use that credential cache for all connections
>>>> instead of $USERNAME and $PASSWORD
>>>>   - create another user if we ever need to modify the groups of the
>>>> administrator (the cached PAC won't update).
>>>>
>>>> I've not got around to doing this yet, but as the python selftest
>>>> rewrite is under way, I wanted to ensure this was catered for in the
>>>> design.
>>>
>>> Thanks.
>>>
>>> I think one of the other issues with selftest is also that we're running
>>> too much high level (functional) tests rather than unit tests. We can't
>>> possibly run all tests with all possible permutations of Samba
>>> configuration options.
>>>
>>> For example, is it useful to run all RPC tests against our own servers
>>> with and without the bigendian option? I can see the bigendian option
>>> being really useful when running tests against Windows, but our client
>>> and server code is generated from the same IDL - we won't find errors in
>>> the IDL this way. If we're trying to catch pidl bugs, I think just
>>> running rpc-echo with and without 'bigendian' should be sufficient, and
>>> more low-level tests for pidl.
>>
>> I agree.  We do test a lot of stuff multiple times, and a lot more is
>> never tested.  We also run almost all the s3 tests with and without
>> encryption.
>>
>> One of my 'pie in the sky' ideas is to match the subunit stream with
>> incremental lcov results to determine which tests are adding coverage,
>> and which tests are just covering the same ground.
>>
>> On a more practical level, if you could get me the command to retrive a
>> time ordered list of tests, it would help me start to attack the slowest
>> tests.
>
> The easiest thing to do here is to get the subunit output for a full test
> run (should be in st/subunit).  You can then feed that into
> "./script/show_test_time" which is a trivial wrapper around "subunit-ls" to
> get a list of test timings. It'll output one line per test with the test
> name and the timing (in seconds).
>
> It might be necessary to split up some tests further, as some of the
> existing tests are pretty big.

May be we can capture the raw output of subunit? This comes in handy
to test subunit framework without having to re-run make test.

http://git.samba.org/?p=amitay/samba.git;a=commitdiff;h=959dc2d0804ec904912a312c8265cf4e2febf947


diff --git a/selftest/wscript b/selftest/wscript
index ab08c62..90056b6 100644
--- a/selftest/wscript
+++ b/selftest/wscript
@@ -186,7 +186,7 @@ def cmd_testonly(opt):
         cmd = '${CORE_COMMAND} --list'
     else:
         env.OPTIONS += ' --socket-wrapper'
-        cmd = '(${CORE_COMMAND} && touch ${SELFTEST_PREFIX}/st_done)
| ${FILTER_OPTIONS}'
+        cmd = '(${CORE_COMMAND} && touch ${SELFTEST_PREFIX}/st_done)
| tee ${SELFTEST_PREFIX}/raw_output.log | ${FILTER_OPTIONS}'
         if (os.environ.get('RUN_FROM_BUILD_FARM') is None and
             not Options.options.FILTERED_SUBUNIT):
             cmd += ' | tee ${SELFTEST_PREFIX}/subunit | ${FORMAT_TEST_OUTPUT}'



Amitay.