Improving the speed of make test

[Jelmer Vernooij]

Am 05/03/12 23:50, schrieb Matthieu Patou:
> On 03/05/2012 12:23 AM, Volker Lendecke wrote:
>> On Mon, Mar 05, 2012 at 06:36:57PM +1100, Andrew Bartlett wrote:
>>> Jelmer and others interested in selftest:
>>>
>>> A while back, I started doing some profiling to determine where the 
>>> time
>>> is spent in 'make test', using 'perf'.
>>>
>>> I was surprised to find that 15% of our time is spent in routines
>> Is that total time or CPU time? Quite some of the tests in
>> the file server area look at or depend on timeout behaviour.
>> The tests for share modes for example come to mind. With
>> some care (independent file names), they could probably very
>> well run in parallel and gain quite a bit of total time.
> I suggested here :https://bugzilla.samba.org/show_bug.cgi?id=8582
>
> that we run tests in environment in parallel (ie. samba3.* tests while 
> doing samba4.* dc  and samba4 fl2000, ...) on machines with more that 
> 2 cores this could prove to be very valuable I'm sure.
>
We actually have the infrastructure for this - you can do most of it 
today using testrepository or with the standard subunit tools and some 
scripting.

Basically, you can list a specific set of tests to run in a file and 
then run those tests by using the --load-list option to "waf selftest". 
For example:

gwenhwyvar:~/src/samba% echo 
samba.tests.xattr.TestCopyTreeWithXattrs.test_simple > todo
gwenhwyvar:~/src/samba% echo 
samba.tests.xattr.XattrTests.test_set_and_get_native >> todo
gwenhwyvar:~/src/samba% ./buildtools/bin/waf test --load-list=todo

The --list option can be used to get a list of all available tests:

gwenhwyvar:~/src/samba% ./buildtools/bin/waf test --list

You can get a list of all failed tests in the previous test run with 
something like:

gwenhwyvar:~/src/samba% subunit-filter --no-passthrough --no-skip 
--no-success < st/subunit | subunit-ls > failed-tests

The result of that command could be used to run all failed tests:

gwenhwyvar:~/src/samba% ./buildtools/bin/waf test --load-list=failed-tests

The main thing that has to be done to be able to use this to be able to 
run subsets of the testsuite (such as all previously failing tests) is 
converting the remaining tests that don't understand the  --list and/or 
--load-list options. selftest will warn about those if you run it with 
--list.

The other thing that's blocking parallel running of the testsuite is the 
fact that environments are in no way shareable at the moment. Thatwill 
hopefully change with my selftest.py work.

Cheers,

Jelmer