How should I structure a Python package that contains Cython code

Question

I\'d like to make a Python package containing some Cython code. I\'ve got the the Cython code working nicely. However, now I want to know how best to package it.

For

Answer 1

This is a setup script I wrote which makes it easier to include nested directories inside the build. One needs to run it from folder within a package.

Givig structure like this:

__init__.py
setup.py
test.py
subdir/
      __init__.py
      anothertest.py

setup.py

from setuptools import setup, Extension
from Cython.Distutils import build_ext
# from os import path
ext_names = (
    'test',
    'subdir.anothertest',       
) 

cmdclass = {'build_ext': build_ext}
# for modules in main dir      
ext_modules = [
    Extension(
        ext,
        [ext + ".py"],            
    ) 
    for ext in ext_names if ext.find('.') < 0] 
# for modules in subdir ONLY ONE LEVEL DOWN!! 
# modify it if you need more !!!
ext_modules += [
    Extension(
        ext,
        ["/".join(ext.split('.')) + ".py"],     
    )
    for ext in ext_names if ext.find('.') > 0]

setup(
    name='name',
    ext_modules=ext_modules,
    cmdclass=cmdclass,
    packages=["base", "base.subdir"],
)
#  Build --------------------------
#  python setup.py build_ext --inplace

Happy compiling ;)

Answer 2

Adding to Craig McQueen's answer: see below for how to override the sdist command to have Cython automatically compile your source files before creating a source distribution.

That way your run no risk of accidentally distributing outdated C sources. It also helps in the case where you have limited control over the distribution process e.g. when automatically creating distributions from continuous integration etc.

from distutils.command.sdist import sdist as _sdist

...

class sdist(_sdist):
    def run(self):
        # Make sure the compiled Cython files in the distribution are up-to-date
        from Cython.Build import cythonize
        cythonize(['cython/mycythonmodule.pyx'])
        _sdist.run(self)
cmdclass['sdist'] = sdist

Answer 3

I've done this myself now, in a Python package simplerandom (BitBucket repo - EDIT: now github) (I don't expect this to be a popular package, but it was a good chance to learn Cython).

This method relies on the fact that building a .pyx file with Cython.Distutils.build_ext (at least with Cython version 0.14) always seems to create a .c file in the same directory as the source .pyx file.

Here is a cut-down version of setup.py which I hope shows the essentials:

from distutils.core import setup
from distutils.extension import Extension

try:
    from Cython.Distutils import build_ext
except ImportError:
    use_cython = False
else:
    use_cython = True

cmdclass = {}
ext_modules = []

if use_cython:
    ext_modules += [
        Extension("mypackage.mycythonmodule", ["cython/mycythonmodule.pyx"]),
    ]
    cmdclass.update({'build_ext': build_ext})
else:
    ext_modules += [
        Extension("mypackage.mycythonmodule", ["cython/mycythonmodule.c"]),
    ]

setup(
    name='mypackage',
    ...
    cmdclass=cmdclass,
    ext_modules=ext_modules,
    ...
)

I also edited MANIFEST.in to ensure that mycythonmodule.c is included in a source distribution (a source distribution that is created with python setup.py sdist):

...
recursive-include cython *
...

I don't commit mycythonmodule.c to version control 'trunk' (or 'default' for Mercurial). When I make a release, I need to remember to do a python setup.py build_ext first, to ensure that mycythonmodule.c is present and up-to-date for the source code distribution. I also make a release branch, and commit the C file into the branch. That way I have a historical record of the C file that was distributed with that release.

Answer 4

The easiest way I found using only setuptools instead of the feature limited distutils is

from setuptools import setup
from setuptools.extension import Extension
try:
    from Cython.Build import cythonize
except ImportError:
    use_cython = False
else:
    use_cython = True

ext_modules = []
if use_cython:
    ext_modules += cythonize('package/cython_module.pyx')
else:
    ext_modules += [Extension('package.cython_module',
                              ['package/cython_modules.c'])]

setup(name='package_name', ext_modules=ext_modules)