How can I write a makefile to auto-detect and parallelize the build with GNU Make?

Question

Not sure if this is possible in one Makefile alone, but I was hoping to write a Makefile in a way such that trying to build any target in the file auto-magically detects the num

Answer 1

I just added this to the top of my Makefile. It lets make create any number of jobs, but tries to keep the load average below the number of cpu cores.

MAKEFLAGS+="-j -l $(shell grep -c ^processor /proc/cpuinfo) "

Note this is Linux specific.

Answer 2

The detection part is going to be OS dependent. Here's a fragment that will work on Linux and Mac OS X:

NPROCS:=1
OS:=$(shell uname -s)

ifeq($(OS),Linux)
  NPROCS:=$(shell grep -c ^processor /proc/cpuinfo)
endif
ifeq($(OS),Darwin) # Assume Mac OS X
  NPROCS:=$(shell system_profiler | awk '/Number Of CPUs/{print $4}{next;}')
endif

To get it working you are probably going to have to re-invoke make. Then your problem is preventing infinite recursion. You could manage that by having two makefiles (the first only resetting the -j value), but it is probably possible to finesse it.

Answer 3

I'll skip over the $(NPROCS) detection stuff, but here's how you could do this in a single Makefile (this is probably GNU Make specific, but that looks like the variant you're running):

ifeq ($(NPROCS),)
# Code to set NPROCS...
%:
    $(MAKE) -j$(NPROCS) NPROCS=$(NPROCS)
else
# All of your targets...
endif

See Defining Last-Resort Default Rules and Overriding Part of Another Makefile in the GNU Make Manual.

Answer 4

After poking around the LDD3 chapter 2 a bit and reading dmckee's answer, I came up with this not so great answer of using two makefiles (I would prefer just one).

$ cat Makefile
MAKEFLAGS += -rR --no-print-directory

NPROCS := 1
OS := $(shell uname)
export NPROCS

ifeq ($J,)

ifeq ($(OS),Linux)
  NPROCS := $(shell grep -c ^processor /proc/cpuinfo)
else ifeq ($(OS),Darwin)
  NPROCS := $(shell system_profiler | awk '/Number of CPUs/ {print $$4}{next;}')
endif # $(OS)

else
  NPROCS := $J
endif # $J

all:
    @echo "running $(NPROCS) jobs..."
    @$(MAKE) -j$(NPROCS) -f Makefile.goals $@

%:
    @echo "building in $(NPROCS) jobs..."
    @$(MAKE) -j$(NPROCS) -f Makefile.goals $@
$ cat Makefile.goals
MAKEFLAGS += -rR --no-print-directory
NPROCS ?= 1

all: subgoal
    @echo "$(MAKELEVEL) nprocs = $(NPROCS)"

subgoal:
    @echo "$(MAKELEVEL) subgoal"

What do you think about this solution?

Benefits I see is that people still type make to build. So there isn't some "driver" script that does the NPROCS and make -j$(NPROCS) work which people will have to know instead of typing make.

Downside is that you'll have to explicitly use make -f Makefile.goals in order to do a serial build. And I'm not sure how to solve this problem...

UPDATED: added $J to above code segment. Seems work work quite well. Even though its two makefiles instead of one, its still quite seamless and useful.

Answer 5

If I read the question correctly, the goal is to parallelize the build process as much as possible. The make man page states the following

If the -j option is given without an argument, make will not limit the number of jobs that can run simultaneously.

Isn't this basically the solution that you want? If your Makefile has enough parallel targets you will use all your CPUs and if the targets are not parallel, that -j option won't help anywas.

Answer 6

If you want it to be automatic, then you can override your typical make command to be an alias to itself in your .bashrc in your home directory.

Example:

alias make="make -j"

Or you could do something like:

alias jmake="make -j"

in the case that you don't want to override it, but want a quick and easy (and memorable) way to run make in parallel.