Supplying NumPy site.cfg arguments to pip

Question

I'm using NumPy built against Intel's Math Kernel Library. I use virtualenv, and typically use pip to install packages.

However, in order for NumPy to find the MKL libraries, it's necessary to create a site.cfg file in the NumPy source directory prior to compiling it, then manually build and install. I could script this whole process, but I was hoping for a simpler solution.

I have a standard site.cfg file that can be used for this purpose under version control. Are there any pip command line options that will tell it to copy a particular file to the source directory before building a package?

Alternatively, are there any environment variables that can be set instead of supplying the library paths in a site.cfg file? Here is the site.cfg file that I use. It was taken almost verbatim from Intel's site.

[mkl]
library_dirs = /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64
include_dirs = /opt/intel/composer_xe_2013.1.117/mkl/include
mkl_libs = mkl_rt
lapack_libs =

For reference, I'm running Ubuntu, Python 2.7, and NumPy 1.6.

Answer 1

From the source (https://github.com/numpy/numpy/blob/master/site.cfg.example):

To assist automatic installation like easy_install, the user's home directory will also be checked for the file ~/.numpy-site.cfg .

Is that a workable solution? You'd still need to preload the home directories with the global .numpy-site.cfg, but you wouldn't have to muck with the build or installation after that.

Answer 2

I ended up putting together a script to automate this. Here it is, in case it can help someone else. I've tested it in Python 2.7, but it should work elsewhere without significant modifications.

from __future__ import unicode_literals

import io
import os.path
import re
import subprocess
import urllib2

# This downloads, builds, and installs NumPy against the MKL in the
# currently active virtualenv

file_name = 'numpy-1.6.2.tar.gz'
url = ('http://sourceforge.net/projects/numpy/files/NumPy/1.6.2/'
       'numpy-1.6.2.tar.gz/download')

def main():

    # download NumPy and unpack it
    file_data = urllib2.urlopen(url).read()
    with io.open(file_name, 'wb') as fobj:
        fobj.write(file_data)
    subprocess.check_call('tar -xvf {0}'.format(file_name), shell=True)
    base_name = re.search(r'(.*)\.tar\.gz$', file_name).group(1)
    os.chdir(base_name)

    # write out a site.cfg file in the build directory
    site_cfg = (
        '[mkl]\n'
        'library_dirs = /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64\n'
        'include_dirs = /opt/intel/composer_xe_2013.1.117/mkl/include\n'
        'mkl_libs = mkl_rt\n'
        'lapack_libs =\n')
    with io.open('site.cfg', 'wt', encoding='UTF-8') as fobj:
        fobj.write(site_cfg)

    # build and install NumPy
    subprocess.check_call('python setup.py build', shell=True)
    subprocess.check_call('python setup.py install', shell=True)


if __name__ == '__main__':
    main()

Answer 3

Your goal of installing NumPy to use Intel's Math Kernel Library is now much easier since Intel created pips to install MKL + NumPy:

pip uninstall numpy -y  # if the standard numpy is present
pip install intel-numpy

as well as intel-scipy, intel-scikit-learn, pydaal, tbb4py, mkl_fft, mkl_random, and the lower level packages if you need just them. Again, you must first uninstall the standard packages if they're already installed in your virtualenv.

NOTE:

If standard NumPy, SciPy and Scikit-Learn packages are already installed, the packages must be uninstalled before installing the Intel® variants of these packages(intel-numpy etc) to avoid any conflicts. As mentioned earlier, pydaal uses intel-numpy, hence it is important to first remove the standard Numpy library (if installed) and then install pydaal.

Answer 4

To your question of how to configure NumPy (e.g. to use OpenBLAS):

1. Download https://github.com/numpy/numpy/blob/master/site.cfg.example
2. Edit the relevant lines, e.g.

[openblas]
libraries = openblas
library_dirs = /opt/OpenBLAS/lib
include_dirs = /opt/OpenBLAS/include

3. Save it as ~/.numpy-site.cfg

4. Install numpy from source without manually downloading it (--force-reinstall will let it replace an existing package):

   pip install numpy --no-binary numpy --force-reinstall

5. Bonus: The same file ~/.numpy-site.cfg works for installing scipy on the OpenBLAS:

   pip install scipy --no-binary scipy

   or install them together:

   pip install numpy scipy --no-binary numpy,scipy --force-reinstall