How to adapt my unit tests to cmake and ctest?

Question

Until now, I\'ve used an improvised unit testing procedure - basically a whole load of unit test programs run automatically by a batch file. Although a lot of these explicitly c

Answer 1

I'd use CMake's standalone scripting mode to run the tests and compare the outputs. Normally for a unit test program, you would write add_test(testname testexecutable), but you may run any command as a test.

If you write a script "runtest.cmake" and execute your unit test program via this, then the runtest.cmake script can do anything it likes - including using the cmake -E compare_files utility. You want something like the following in your CMakeLists.txt file:

enable_testing()
add_executable(testprog main.c)
add_test(NAME runtestprog
    COMMAND ${CMAKE_COMMAND}
    -DTEST_PROG=$
    -DSOURCEDIR=${CMAKE_CURRENT_SOURCE_DIR}
    -P ${CMAKE_CURRENT_SOURCE_DIR}/runtest.cmake)

This runs a script (cmake -P runtest.cmake) and defines 2 variables: TEST_PROG, set to the path of the test executable, and SOURCEDIR, set to the current source directory. You need the first to know which program to run, the second to know where to find the expected test result files. The contents of runtest.cmake would be:

execute_process(COMMAND ${TEST_PROG}
                RESULT_VARIABLE HAD_ERROR)
if(HAD_ERROR)
    message(FATAL_ERROR "Test failed")
endif()

execute_process(COMMAND ${CMAKE_COMMAND} -E compare_files
    output.txt ${SOURCEDIR}/expected.txt
    RESULT_VARIABLE DIFFERENT)
if(DIFFERENT)
    message(FATAL_ERROR "Test failed - files differ")
endif()

The first execute_process runs the test program, which will write out "output.txt". If that works, then the next execute_process effectively runs cmake -E compare_files output.txt expected.txt. The file "expected.txt" is the known good result in your source tree. If there are differences, it errors out so you can see the failed test.

What this doesn't do is print out the differences; CMake doesn't have a full "diff" implementation hidden away within it. At the moment you use Subversion to see what lines have changed, so an obvious solution is to change the last part to:

if(DIFFERENT)
    configure_file(output.txt ${SOURCEDIR}/expected.txt COPYONLY)
    execute_process(COMMAND svn diff ${SOURCEDIR}/expected.txt)
    message(FATAL_ERROR "Test failed - files differ")
endif()

This overwrites the source tree with the build output on failure then runs svn diff on it. The problem is that you shouldn't really go changing the source tree in this way. When you run the test a second time, it passes! A better way is to install some visual diff tool and run that on your output and expected file.