Automatically setting jobs (-j) flag for a multicore machine?

Question

I have a Makefile on a machine that has a ton of cores in it, but I always seem to forget to write -jX when compiling my project and it takes way longer than it

Answer 1

At the beginning of a Makefile:

MAKEFLAGS+="j"

It won't take numeric arguments for any version of GNU Make before 4.2. After 4.2 you can do:

MAKEFLAGS+="j2"

For versions earlier than 4.2 if you happen to have some jobs running out of memory, you could make them run only one at a time with flock from util-linux-ng. For example, a convert utility from ImageMagick will use all resources it can get when used to optimize images according to Google's recommendations, so there's no point to have it run in parallel.

%.min.jpg: %.jpg
    @flock --wait 600 Makefile convert $< -sampling-factor 4:2:0 -strip $@

It is important to set a long wait time because make will still run most of these commands in parallel. Therefore, wait time must be as such as the deepest queue execution time. If you have eight cores, and eight large images take a minute to optimize, you must set the wait time to at least a minute.

With hundreds of cores and huge images, six hundred seconds set above might not be enough.