How should I structure a Python package that contains Cython code

Question

I\'d like to make a Python package containing some Cython code. I\'ve got the the Cython code working nicely. However, now I want to know how best to package it.

For

Answer 1

All other answers either rely on

• distutils
• importing from Cython.Build, which creates a chicken-and-egg problem between requiring cython via setup_requires and importing it.

A modern solution is to use setuptools instead, see this answer (automatic handling of Cython extensions requires setuptools 18.0, i.e., it's available for many years already). A modern standard setup.py with requirements handling, an entry point, and a cython module could look like this:

from setuptools import setup, Extension

with open('requirements.txt') as f:
    requirements = f.read().splitlines()

setup(
    name='MyPackage',
    install_requires=requirements,
    setup_requires=[
        'setuptools>=18.0',  # automatically handles Cython extensions
        'cython>=0.28.4',
    ],
    entry_points={
        'console_scripts': [
            'mymain = mypackage.main:main',
        ],
    },
    ext_modules=[
        Extension(
            'mypackage.my_cython_module',
            sources=['mypackage/my_cython_module.pyx'],
        ),
    ],
)

Answer 2

The easiest is to include both but just use the c-file? Including the .pyx file is nice, but it's not needed once you have the .c file anyway. People who want to recompile the .pyx can install Pyrex and do it manually.

Otherwise you need to have a custom build_ext command for distutils that builds the C file first. Cython already includes one. http://docs.cython.org/src/userguide/source_files_and_compilation.html

What that documentation doesn't do is say how to make this conditional, but

try:
     from Cython.distutils import build_ext
except ImportError:
     from distutils.command import build_ext

Should handle it.

Answer 3

http://docs.cython.org/en/latest/src/userguide/source_files_and_compilation.html#distributing-cython-modules

It is strongly recommended that you distribute the generated .c files as well as your Cython sources, so that users can install your module without needing to have Cython available.

It is also recommended that Cython compilation not be enabled by default in the version you distribute. Even if the user has Cython installed, he probably doesn’t want to use it just to install your module. Also, the version he has may not be the same one you used, and may not compile your sources correctly.

This simply means that the setup.py file that you ship with will just be a normal distutils file on the generated .c files, for the basic example we would have instead:

from distutils.core import setup
from distutils.extension import Extension
 
setup(
    ext_modules = [Extension("example", ["example.c"])]
)

Answer 4

I think I found a pretty good way of doing this by providing a custom build_ext command. The idea is the following:

1. I add the numpy headers by overriding finalize_options() and doing import numpy in the body of the function, which nicely avoids the problem of numpy not being available before setup() installs it.

2. If cython is available on the system, it hooks into the command's check_extensions_list() method and by cythonizes all out-of-date cython modules, replacing them with C extensions that can later handled by the build_extension() method. We just provide the latter part of the functionality in our module too: this means that if cython is not available but we have a C extension present, it still works, which allows you to do source distributions.

Here's the code:

import re, sys, os.path
from distutils import dep_util, log
from setuptools.command.build_ext import build_ext

try:
    import Cython.Build
    HAVE_CYTHON = True
except ImportError:
    HAVE_CYTHON = False

class BuildExtWithNumpy(build_ext):
    def check_cython(self, ext):
        c_sources = []
        for fname in ext.sources:
            cname, matches = re.subn(r"(?i)\.pyx$", ".c", fname, 1)
            c_sources.append(cname)
            if matches and dep_util.newer(fname, cname):
                if HAVE_CYTHON:
                    return ext
                raise RuntimeError("Cython and C module unavailable")
        ext.sources = c_sources
        return ext

    def check_extensions_list(self, extensions):
        extensions = [self.check_cython(ext) for ext in extensions]
        return build_ext.check_extensions_list(self, extensions)

    def finalize_options(self):
        import numpy as np
        build_ext.finalize_options(self)
        self.include_dirs.append(np.get_include())

This allows one to just write the setup() arguments without worrying about imports and whether one has cython available:

setup(
    # ...
    ext_modules=[Extension("_my_fast_thing", ["src/_my_fast_thing.pyx"])],
    setup_requires=['numpy'],
    cmdclass={'build_ext': BuildExtWithNumpy}
    )

Answer 5

Including (Cython) generated .c files are pretty weird. Especially when we include that in git. I'd prefer to use setuptools_cython. When Cython is not available, it will build an egg which has built-in Cython environment, and then build your code using the egg.

A possible example: https://github.com/douban/greenify/blob/master/setup.py

---

Update(2017-01-05):

Since setuptools 18.0, there's no need to use setuptools_cython. Here is an example to build Cython project from scratch without setuptools_cython.

Answer 6

The simple hack I came up with:

from distutils.core import setup

try:
    from Cython.Build import cythonize
except ImportError:
    from pip import pip

    pip.main(['install', 'cython'])

    from Cython.Build import cythonize


setup(…)

Just install Cython if it could not be imported. One should probably not share this code, but for my own dependencies it's good enough.