CMake Proposal

[Bill Hoffman]

tridge at samba.org wrote:

> The other really big problem I have with the cmake proposal is that
> the people proposing it don't seem to have taken the time to look at
> our existing Samba build system to see what it is they are trying to
> replace. The existing build system has problems, but is also has some
> great features. You can't seriously expect to replace it without
> understanding it first. That means diving into the config.mk files and
> understanding what they are and how they work. It also means
> understanding how it integrates with subunit testing, and how things
> like 'make testenv' are implemented.
> 
I just took a look at the testing stuff in the samba build tree.  To 
make the tests work in a cross platform way (not requiring /bin/sh), 
would take a little more work, but not much.  CMake has a strong test 
infrastructure.   Anyway, I did not test the following, but it should 
give you and idea of what running tests in CMake might look like. I may 
have missed something in how the samba tests work, as I did not run 
them, but just looked at the config.mk file.   It seemed that the 
testenv run the same test perl script, but with these options: 
--socket-wrapper --testenv

Given that assumption, here is what I came up with:


# This will set SAMBA_SOURCE_DIR and SAMBA_BINARY_DIR
# I am assuming this is done at the top of the whole
# samba source tree and not in the test subdir

project(SAMBA C)
find_package(Perl REQURIED) # will set PERL_EXECUTABLE
find_package(PythonInterp REQURIED) # will set PYTHON_EXECUTABLE


# this would go into the CMakeLists.txt file in
# source4/selftest/CMakeLists.txt


set(DEFAULT_TEST_OPTIONS ""  CACHE STRING
"Default options for selftest.pl")
set(SELFTEST_RUN_DIR "${SAMBA_BINARY_DIR}/st" CACHE PATH
"Path to directory for running tests")
set(TEST_FORMAT "plain" CACHE STRING "Default test format")

# create a function to add a cmake style test
# This will run selftest.pl and pass it a regular expression
# that will run one test
# Arguments:
#   name  - The test name
#   test  - The test name passed to selftest

function(add_samba_test name test)
   add_test(${name}
     ${PERL_EXECUTABLE} ${SAMBA_SOURCE_DIR}/selftest/selftest.pl
     --prefix=${SELFTEST_RUN_DIR}
     --builddir=${SAMBA_BINARY_DIR}
     --srcdir=${SAMBA_SOURCE_DIR}
     --expected-failures=${SAMBA_SOURCE_DIR}/selftest/knowfail
     --format=${TEST_FORMAT}
     --exclude=${SAMBA_SOURCE_DIR}/selftest/skip
     --testlist=${ALL_TESTS}
     "^${test}$"
     )
   set_tests_properties(${name} PROPERTIES
     ENVIRONMENT "PYTHON=${PYTHON_EXECUTABLE}")
endfunction(add_samba_test)

# run the tests.sh script to generate the list of tests
# To be more portable this could just be converted to a cmake
# variable, or the tests could be just listed in this file.
# As someone that cares about things running on windows using
# /bin/sh worries me....  :)

execute_process(COMMAND /bin/sh ${CMAKE_CURRENT_SOURCE_DIR}/tests.sh
   OUTPUT_VARIABLE ALL_TESTS)

# loop over all the tests and create a cmake test for each test
foreach(test ${ALL_TESTS})
   # create the regular test
   add_samba_test("${test}" "${test}" )
   # also create a -env test, this will run the same test, but
   # with the --socket-wrapper and the --testenv options
   add_samba_test("${test}-env" "${test}" --socket-wrapper --testenv)
endforeach()


# to run the tests from the command line you would do this:

# run all the tests:
ctest
make test

# run tests that match a regex
ctest -R regex
# exclude tests that match a regex
ctest -E regex

# run the tests in parallel 10 at a time
ctest -j 10

# the following would build samba, run the tests and
# submit the results to a cdash dashbard
# see www.cdash.org
ctest -D Experimental

ctest -N Show all the tests but do not run them.


In the long run, I think even selftest.pl itself could be replaced by 
the native CMake/CTest commands.   We have been working with the 
Trilinos group at Sandia to enhance CTest/CDash to handle running many 
different kinds of tests. Here is a link to that dashboard:

http://trilinos.sandia.gov/cdash/


If CMake does get adopted by Samba I can help setup and tune the testing 
part.

-Bill