tensorflow gpu can not be called from jupyterhub/jupyter notebook, why?

Question

Well I figure eight hours is enough time trying to fix this on my own, so I'll just ask folks:

I am running tensorflow-gpu 1.1.0 just fine in my virtual environment named 'tensorflow' outside of jupyterhub and Jupyter notebook. That is, I can import tensorflow and run scripts with it using the gpu.

When I'm inside my tensorflow virtualenv and using jupyterhub, Jupyter can not seem to 'see' tensorflow. I get the following error:

ImportError: libcublas.so.8.0: cannot open shared object file: No such file or directory

1) This is a common seen error message indicating tensorflow install problems, yet my paths and environment variables seem fine. After all, I can use tensorflow-gpu just fine outside of Jupyter.

2) Typing 'which jupyter' shows ~/anaconda3/envs/hub/bin/jupyter, so I believe that I am referencing jupyter inside my virtualenv. 3) Pip freeze shows that I have jupyterhub and tensorflow-gpu. I even did a pip3 freeze and it shows both packages as well.

Any ideas? Can tensorflow-gpu be run from Jupyter notebooks?

Answer 1

I got the solution from here:

[https://github.com/jupyter/notebook/issues/1290][1]

Basically, something was 'wrong' with jupyter in that it could not read my LD_LIBRARY_PATH variable. I did put everything correctly inside .bashrc so I don't know why.

Switch to the command line (terminal). Switch into your virtual environment if you have one.

type in: jupyter notebook --generate-config

It will tell you the directory in which your jupyter configuration file is stored. If you want to list it again type: jupyter --config-dir

Mine jupyter_notebook_config.py file is located here: /home/me/.jupyter/jupyter_notebook_config.py

At the very top of this file, jupyter_notebook_config.py, add in the following code:

import os
c = get_config()
os.environ['LD_LIBRARY_PATH'] = '/usr/local/cuda-8.0/lib64:usr/local/cuda-8.0/lib64/libcudart.so.8.0'
c.Spawner.env.update('LD_LIBRARY_PATH')

Then restart jupyterhub or jupyter notebook (type in at the command line: jupyter notebook

Tensorflow gpu should work.

The same thing applies even if you are running jupyterhub. Make the changes in jupyter, not jupyterhub. (Each user of jupyterhub will have their own jupyter process, so make the changes not at the 'hub' level, but the jupyter notebook level.