Distributing a shared library and some C code with a Cython extension module

Question

I\'m trying to take some functions from a large C++ shared library (libbig.so) and expose them to Python via Cython. To do so, I\'ve got a little C++ file (small.cpp) that provi

Answer 1

Here's my tricky solution. The idea is to "hide" the presence of cython until it's installed by the requirements. This can be achieved by lazy evaluation. Here's an example:

from setuptools import setup, Extension

class lazy_cythonize(list):
    def __init__(self, callback):
        self._list, self.callback = None, callback
    def c_list(self):
        if self._list is None: self._list = self.callback()
        return self._list
    def __iter__(self):
        for e in self.c_list(): yield e
    def __getitem__(self, ii): return self.c_list()[ii]
    def __len__(self): return len(self.c_list())

def extensions():
    from Cython.Build import cythonize
    ext = Extension('native_ext_name', ['your/src/*.pyx'])
    return cythonize([ext])


configuration = {
    'name': 'mypackage',
    'packages': ['yourpurepythonpackage'],
    'install_requires': ['cython==0.17'],
    'ext_modules': lazy_cythonize(extensions)
}

setup(**configuration)

lazy_cythonize is a fake list that generates its internal elements only when someone tries to access to it.
When it's required, this class imports Cython.Build and generates the list of extensions. This avoids to keep the *.c files in your project, requiring cython to be installed when the module is building.

Quite tricky, but actually it's working.

Answer 2

I've pushed a fix for setuptools 288, slated for release as setuptools 18.0. This changelog entry describes a technique that should work with that build. A beta release is available for testing.

Answer 3

1) Distributing libbig.so

This is a problem that python isn't going to help you with. Who are you targeting? If it's linux, can you request that they install it with their package manager? If libbig isn't distributed through a package manager or it's not linux and you're targeting multiple architectures, you might have to distribute the libbig source.

2) Cython/setuptools.

Frankly, I think its easiest to just require that people have Cython. This way there's only one ground truth version of the code, and you don't have to worry about inconsistencies between the .pyx and .cpp code. The easiest way to do this is to use setuptools instead of distutils. That way, you can use:

setup('mypackage',
    ...
    install_requires=['cython'])

In total, your setup.py script will look something like:

# setup.py

from setuptools import setup, Extension
from Cython.Distutils import build_ext

pysmall = Extension('pysmall',
    sources = ['pysmall.pyx', 'small.cpp'],
    include_dirs = ['include/'])

setup(name='mypackage',
      packages=['yourpurepythonpackage'],
      install_requires=['cython==0.17'],
      ext_modules=[pysmall],
      cmdclass = {'build_ext': build_ext})

If you don't like the idea of requiring cython, you could do something like:

# setup.py

import warnings
try:
    from Cython.Distutils import build_ext
    from setuptools import setup, Extension
    HAVE_CYTHON = True
except ImportError as e:
    HAVE_CYTHON = False
    warnings.warn(e.message)
    from distutils.core import setup, Extension
    from distutils.command import build_ext

pysmall = Extension('pysmall',
    sources = ['pysmall.pyx', 'small.cpp'],
    include_dirs = ['include/'])

configuration = {'name': 'mypackage',
      'packages': ['yourpurepythonpackage'],
      'install_requires': ['cython==0.17'],
      'ext_modules': [pysmall],
      'cmdclass': {'build_ext': build_ext}}

if not HAVE_CYTHON:
    pysmall.sources[0] = 'pysmall.cpp'
    configuration.pop('install_requires')

setup(**configuration)