1. 技术文章
  2. 'cmake rebuild_cache' for *just* a subdirectory?

'cmake rebuild_cache' for *just* a subdirectory?

由 寵の児 提交于 2019-11-29 04:23:51
Florian

There are so many aspects that define CMake's configuration and generation steps duration (besides what you actually do in your CMakeLists.txt files; it's e.g. your host system, your toolchain and which CMake version/distribution you are using).

So I try to concentrate on the specific questions you have.

Rebuild/rewrite makefiles for just a subdirectory?

For the start: Using add_subdirectory() is good for structuring your CMake code. But you have to keep in mind that you can always change global CMake properties in a subdirectory and that targets inside those subdirectories can have cross-dependencies.

So what does CMake do (considering the "I have touched one CMakeLists.txt file in a subdirectory" case discussed here):

  • If a CMakeLists.txt file is changed it goes through the complete hierarchy of CMakeLists.txt files again and rebuilds the build environment again in memory.
  • Now it temporarily recreates all the necessary build/make files and checks if they defer from the existing ones (see cmGeneratedFileStreamBase::Close()).
  • If a file has changed it replaces the existing one with the new one.

This behavior is necessary because any makefile can change even when only a subdirectory's CMakeLists.txt file has changed and it was optimized to prevent unnecessary rebuilds during the actual make step (from touched makefiles).

Is there some way to speed up the generation step of CMake?

So yes, it does temporarily rewrite all makefiles (which could be slow) and no, you can't minimize this while using add_subdirectory() to only the changed subdirectory.

Maybe one possible performance optimization for the future in CMake's own code would be to use memorystreams instead of filestreams for the temporary files.

@BruceAdams tested this by using a RAM disk for the generated makefile environment with no effect.

And yes, the CMake generated cmake_check_build_system rule does almost the same as the rebuild_cache rule and yes the used -B, -H and --check-build-system options are CMake internal command line options and therefore undocumented (even if often referred to on Stack Overflow, e.g. in one of my answers here).

What helped me to speed-up the configuration/generation was to rebuild CMake itself with a lot more optimization options than the normal distributions and using a 64-bit toolchain instead of the 32-bit versions currently still distributed.

Here are some test results (using the CMake test script found below with 100 subdirectories/libraries) on my Windows PC always using the same MSYS environment, but different CMake compilations of the same CMake source code:

  1. Official CMake 3.2.2 version:

    $ time -p cmake -G "MSYS Makefiles" ..
[...]
real 43.93
user 0.00
sys 0.03

  2. Using mingw32 and GNU 4.8.1 I rebuild CMake 3.2.2 with

    cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS="-O3" -G "MSYS Makefiles" ..

    and got

    $ time -p /c/temp/cmake-3.2.2/MSYS32/bin/cmake.exe -G "MSYS Makefiles" ..
[...]
real 41.37
user 0.01
sys 0.04

    And the same with my antivirus software turned off:

    $ time -p /c/temp/cmake-3.2.2/MSYS32/bin/cmake.exe -G "MSYS Makefiles" ..
[...]
real 20.98
user 0.00
sys 0.04

  3. Using mingw-w64 and GNU 5.3.0 I rebuild CMake 3.2.2 with

    $ cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS="-march=native -m64 -Ofast  -flto" -G "MSYS Makefiles" ..

    and got

    $ time -p /c/temp/cmake-3.2.2/MSYS64/bin/cmake.exe -G "MSYS Makefiles" ..
[...]
real 25.59
user 0.00
sys 0.04

    And the same with my antivirus software turned off:

    $ time -p /c/temp/cmake-3.2.2/MSYS64/bin/cmake.exe -G "MSYS Makefiles" ..
[...]
real 6.95
user 0.00
sys 0.03

To summarize I see two main influences:

1st: The configuration step can be speed-up by going for a 64-bit version and optimize for your processor platform (you would certainly have to find a common base -march=... or -mtune=... for all your project's build PCs).

2nd: The generation step can mostly be sped up by searching for possible file I/O bottlenecks outside of CMake. In my case telling the antivirus software to not check the toolchain and build directories each time I read/write to those was really speeding up things.

Remark: I confirmed @BruceAdams test results that the the compiler's auto-vectorization (default for -O3 or -Ofast) is not able to do much about CMake source code's ability to run in multiple processes/on multiple cores.

Is there a better way to structure a CMake project to achieving this?

Yes, if you e.g. know that a certain sub-tree of your CMake script code just generates a library and has no dependencies, you could put that part in an external project by using ExternalProject_Add(). And yes, having had similar concerns regarding large CMake projects, this is seen as a good "modern CMake" practice (see also references below).

References

What I used to reproduce your problem

Just for completeness and if someone wants to check those numbers against his/her own, here is my test code:

cmake_minimum_required(VERSION 3.0)

project(CMakeTest CXX)

#set_property(GLOBAL PROPERTY GLOBAL_DEPENDS_DEBUG_MODE 1)

set(_idx 1)

while (_idx LESS 100)
    math(EXPR _next_idx "${_idx} + 1")
    if (NOT EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/lib${_idx}")
        file(MAKE_DIRECTORY "lib${_idx}")
        file(
            WRITE "lib${_idx}/lib${_idx}.h"
                "int lib${_idx}_func();"
        )
        file(
            WRITE "lib${_idx}/lib${_idx}.cc"
                "#include \"lib${_next_idx}.h\"\n"
                "int lib${_idx}_func() { return lib${_next_idx}_func(); }"
        )
        file(
            WRITE "lib${_idx}/CMakeLists.txt"
                "add_library(lib${_idx} \"lib${_idx}.cc\")\n"
                "target_link_libraries(lib${_idx} lib${_next_idx})\n"
                "target_include_directories(lib${_idx} PUBLIC \".\")"
        )
    endif()
    add_subdirectory("lib${_idx}")
    set(_idx "${_next_idx}")
endwhile()

if (NOT EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/lib${_idx}")
    file(MAKE_DIRECTORY "lib${_idx}")
    file(
        WRITE "lib${_idx}/lib${_idx}.h"
            "int lib${_idx}_func();"
    )
    file(
        WRITE "lib${_idx}/lib${_idx}.cc"
            "int lib${_idx}_func() { return 0; }"
    )
    file(
        WRITE "lib${_idx}/CMakeLists.txt"
            "add_library(lib${_idx} \"lib${_idx}.cc\")\n"
            "target_include_directories(lib${_idx} PUBLIC \".\")"
    )
endif()
add_subdirectory("lib${_idx}")

if (NOT EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/main.cc")
    file(
        WRITE "main.cc"
            "#include \"lib1.h\"\n"
            "int main() { return lib1_func(); }"
    )
endif()

add_executable(${PROJECT_NAME} "main.cc")
target_link_libraries(${PROJECT_NAME} lib1)

And then - after the first cmake .. and make calls - doing:

$ touch ../lib100/CMakeLists.txt
$ time -p cmake ..
-- Configuring done
-- Generating done
-- Build files have been written to: [your path here]
real 28.89
user 0.01
sys 0.04
标签
build
cmake
build-process
meson-build
易学教程内所有资源均来自网络或用户发布的内容，如有违反法律规定的内容欢迎反馈！
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!