Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so

Question

I am running a python script and I get this error:

Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.

Both files are present

Answer 1

I had this same issue using scikit-learn 0.19 and numpy 1.13.3 when running MLPRegressor (and also with a package called pyearth running an algorithm called MARS). I believe the root of the problem was that our python is part of an Anaconda install, but scikit-learn and numpy were installed via pip, and their expectations for mkl must not agree.

Unfortunately my framework is managed by some dedicated company admins, not by me, so I haven't gotten my guy to try recompiling numpy yet. But I was able to find a workaround based on this thread: Adding export LD_PRELOAD=/path/to/anaconda/lib/libmkl_def.so:/path/to/anaconda/lib/libmkl_avx.so:/path/to/anaconda/lib/libmkl_core.so:/path/to/anaconda/lib/libmkl_intel_lp64.so:/path/to/anaconda/lib/libmkl_intel_thread.so:/path/to/anaconda/lib/libiomp5.so to my ~/.bashrc causes the problem to disappear. It's super hacky, and I'd be lying if I said I knew exactly what it's doing (but this is helpful), so I'm hoping a recompile of numpy is a cleaner fix. But at least it works.

Note it is better performance-wise to have the versions of these packages that use mkl. Installing the nomkl versions is a workaround but not a true solution.