Installing a pip package with cupy as a requirement puts install in never ending loop

问题

I am trying to make a pip package with cupy as one of the requirements, but I include cupy in the requirement, the pip install ends up in a never ending loop. I am trying to install the package on Google Colab, which already has Cupy install, so it should only check if Cupy is already installed and not try to install it again.

I made a minimal pip package in github where cupy is the only requirement.

https://github.com/Santosh-Gupta/TroubleShootCupyInstall

I tried to install it in Google Colab with

!pip install --verbose https://github.com/Santosh-Gupta/TroubleShootCupyInstall/archive/master.zip --log 'file.log'

The output is a lot since it is in verbose, but these are the lines which are printed when it gets to its loop.

 x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -D_FORCE_INLINES=1 -I/usr/local/cuda/include -I/usr/include/python3.6m -c cupy/cuda/nvtx.cpp -o build/temp.linux-x86_64-3.6/cupy/cuda/nvtx.o
  x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.6/cupy/cuda/nvtx.o -L/usr/local/cuda/lib64 -lnvToolsExt -o build/lib.linux-x86_64-3.6/cupy/cuda/nvtx.cpython-36m-x86_64-linux-gnu.so -Wl,--disable-new-dtags,-rpath,/usr/local/cuda/lib64
  building 'cupy.cuda.thrust' extension
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -D_FORCE_INLINES=1 -I/usr/local/cuda/include -I/usr/include/python3.6m -c cupy/cuda/thrust.cpp -o build/temp.linux-x86_64-3.6/cupy/cuda/thrust.o
  NVCC options: ['--generate-code=arch=compute_30,code=compute_30', '--generate-code=arch=compute_50,code=compute_50', '--generate-code=arch=compute_60,code=sm_60', '--generate-code=arch=compute_61,code=sm_61', '--generate-code=arch=compute_70,code=sm_70', '--generate-code=arch=compute_75,code=sm_75', '--generate-code=arch=compute_70,code=compute_70', '-O2', '--compiler-options="-fPIC"']
  /usr/local/cuda/bin/nvcc -D_FORCE_INLINES=1 -I/usr/local/cuda/include -I/usr/include/python3.6m -c cupy/cuda/cupy_thrust.cu -o build/temp.linux-x86_64-3.6/cupy/cuda/cupy_thrust.o --generate-code=arch=compute_30,code=compute_30 --generate-code=arch=compute_50,code=compute_50 --generate-code=arch=compute_60,code=sm_60 --generate-code=arch=compute_61,code=sm_61 --generate-code=arch=compute_70,code=sm_70 --generate-code=arch=compute_75,code=sm_75 --generate-code=arch=compute_70,code=compute_70 -O2 --compiler-options="-fPIC"

For convenience I made a Google Colab notebook which runs this line, and has the full output.

https://colab.research.google.com/drive/1DFR78cJ07KaHkJfpjh8370SxNw0HXI50

回答1:

CuPy currently provides source package named cupy and binary distribution packages named cupy-cudaXX (where XX is a CUDA version). Currently Google Colab is shipped with cupy-cuda100 because it is using CUDA 10.0. If you specify cupy as an requirement of your package, cupy source package will be downloaded and installed (requires build which takes several minutes), even though CuPy is already available through cupy-cuda100.

Unfortunately, Python package distribution tools (such as setuptools, pip, etc.) do not provide a way to well-handle this kind of complex package composition.

Workaround 1

In setup.py (or in your package's __init__.py)

try:
  import cupy
except Exception:
  raise RuntimeError('CuPy is not available. Please install it manually: https://docs-cupy.chainer.org/en/stable/install.html#install-cupy')

# You can also use `cupy.__version__` and `numpy.lib.NumpyVersion` to test CuPy version requirement here.

Workaround 2

Manually check requirements using pkg_resources (part of setuptools) as done in Chainer.

https://github.com/chainer/chainer/blob/v6.3.0/chainer/_environment_check.py#L44
https://github.com/chainer/chainer/blob/v6.3.0/chainer/_version.py#L4-L18

来源：https://stackoverflow.com/questions/57721158/installing-a-pip-package-with-cupy-as-a-requirement-puts-install-in-never-ending

标签

python

pip

google-colaboratory

cupy