Automatically setting jobs (-j) flag for a multicore machine?

Question

I have a Makefile on a machine that has a ton of cores in it, but I always seem to forget to write -jX when compiling my project and it takes way longer than it

Answer 1

At the beginning of a Makefile:

MAKEFLAGS+="j"

It won't take numeric arguments for any version of GNU Make before 4.2. After 4.2 you can do:

MAKEFLAGS+="j2"

For versions earlier than 4.2 if you happen to have some jobs running out of memory, you could make them run only one at a time with flock from util-linux-ng. For example, a convert utility from ImageMagick will use all resources it can get when used to optimize images according to Google's recommendations, so there's no point to have it run in parallel.

%.min.jpg: %.jpg
    @flock --wait 600 Makefile convert $< -sampling-factor 4:2:0 -strip $@

It is important to set a long wait time because make will still run most of these commands in parallel. Therefore, wait time must be as such as the deepest queue execution time. If you have eight cores, and eight large images take a minute to optimize, you must set the wait time to at least a minute.

With hundreds of cores and huge images, six hundred seconds set above might not be enough.

Answer 2

I'm assuming you're using Linux. This is from my ~/.bashrc

# parallel make
export NUMCPUS=`grep -c '^processor' /proc/cpuinfo`
alias pmake='time nice make -j$NUMCPUS --load-average=$NUMCPUS'

sample usage

samm@host src> echo $NUMCPUS
8
samm@host src> pmake

becomes time nice make -j8 --load-average=8.

To answer your specific question about putting this into a Makefile, I don't find it practical to sprinkle this logic all over my Makefiles. Putting this into a top level Makefile also isn't a great solution since I often build from sub-directories and wish to build them in parallel as well. However, if you have a fairly flat source hierarchy, it may work for you.

Answer 3

You can add a line to your Makefile similar to the following:

NUMJOBS=${NUMJOBS:-" -j4 "}

Then add a ${NUMJOBS} line in your rules, or add it into another Makefile var (like MAKEFLAGS). This will use the NUMJOBS envvar, if it exists; if it doesn't, automatically use -j4. You can tune or rename it to your taste.

(N.B.: Personally, I'd prefer the default to be -j1 or "", especially if it's going to be distributed to others, because although I have multiple cores also, I compile on many different platforms, and often forget to dis-able the -jX setting.)

Answer 4

On Ubuntu 16.4 using all CPU cores:

export MAKEFLAGS='-j$(nproc)'

or

export MAKEFLAGS='-j 2'