makefile with directory tree creation suitable for parallel (-j ) build

Question

My project needs temporary directories which are created during the build using mkdir -p similarly to this:

all: dirtree $(OBJFILES)

dirtree: 
  @mkdir -p $         


        

Answer 1

The problem with your makefile is that creation of your object files does not depend on creation of the relevant directories (only a phony "all" target does). This kind of dependency is necessary for -j option, and even without it your makefile works only by chance. There are two (right) ways to impose the dependency in question.

Directories as separate targets

You created the target for directory creation; what left is just put it as a prerequisite to object file rule:

$(BUILD)/temp_directory/%.o: %.c   |   dirtree
        $(CC) $^ -o $@

The pipe symbol | means that dirtree is an "order only prerequisite". It is used when "dirtree" is a prerequisite but changes in the dirtree do not invalidate object files and do not affect the outcome of compilation command.

Use of "order-only" prerequisite is important here. The thing is that dirtree target would be remade at each Make invocation. That would cause everything that depends on it be remade as well, so it would rebuild all object files every time.

Create directories in shell commands

Another way is to ensure that the directory is created immediately before you invoke compilation

$(BUILD)/temp_directory/%.o: %.c
        @mkdir -p $(@D)
        $(CC) $^ -o $@

Note the usage of $(@D). This is expanded as "the directory for the target file". So it may be used uniformly in many places, and even with aid of a variable.

Mkdir=@mkdir -p $(@D)
$(BUILD)/temp_directory/%.o: %.c
        $(Mkdir)
        $(CC) $^ -o $@
$(INSTALL_DIR)/%: src_dir/%
        $(Mkdir)
        cp -p $^ $@

---

Both ways ensure that the directory is created before the compilation commands are invoked. Both ways require you to write some text (either | dirtree or $(Mkdir)) at each rule that needs it. Both ways are -j compatible, but the second solution requires mkdir -p to be thread-safe (as two such commands at once may try to create the same directory, and one of them would fail).

While most systems implement it in such a way that mkdir -p is more or less thread safe, on some systems (as in some Solaris systems, for example), they are less thread-safe than the others. However, even in GNU toolchain mkdir -p may fail if they simultaneously invoke the same mkdir(2) library call.

If you want to be very safe, you can work this around as well. What could be the problem? That two mkdir -p scripts try to create the same directory, and clash somewhere inside C library. Then, one of these mkdir-s will succeed, and the other will fail. However, if the mkdir you invoked failed, then it could be thread-unsafety-related failure only if the directory had been created by a concurrent mkdir. So it would be enough to just check that the target directory is created after mkdir invocation:

Mkdir=@mkdir -p $(@D) || test -d $(@D)

(This solution also has an issue with mode: mkdir may fail when directory exists, but doesn't conform to umask, so you might want to check that as well. But that's too much I guess.)

Answer 2

I'm not sure I fully understand your question. However, I can say this: if your build breaks when you add parallelism, then it's an indication that you haven't defined the dependencies correctly. Ask yourself, "Do the directories need to exist before the object files are generated?" If the answer is "yes", then the directories should be listed as prerequisites of the object files. In other words:

${OBJFILES}: dirtree

And yes, that is pretty much the standard way to do this :)

Answer 3

You could have the rules for building the object files call mkdir -p as their first action.