uWSGI and joblib Semaphore: Joblib will operate in serial mode

Question

I\'m running joblib in a Flask application living inside a Docker container together with uWSGI (started with threads enabled) which is started by supervisord.

The start

Answer 1

This was quite a rabbit hole.

The joblib issues page on Github has similar posts of joblib failing with Uwsgi. But most are for the older multiprocessing backend. The new loky backend was supposed to solve these issues.

There was PR for the multiprocessing backend that solved this issue for uwsgi:

joblib.Parallel(n_jobs=4,backend="multiprocessing")(joblib.delayed(sqrt)(i ** 2) for i in range(10))

But it failed sometimes randomly and fell back to the same issue that the PR above tried to solve.

Further digging revealed that the present backend loky parallelizes on processes by default (docs). But these processes dont have shared memory access and so need serialized and queued channels. This is probably the reason why uWSGI fails and gunicorn works.

So I tried switching to threads instead of processes:

joblib.Parallel(n_jobs=4,prefer="threads")(joblib.delayed(sqrt)(i ** 2) for i in range(10))

And it works :)