How to include external library with python wheel package

Question

I want to create package for python that embeds and uses an external library (.so) on Linux using the cffi module.

Is there standard way to include .so file into python package?

The package will be used only internally and won't be published to pypi.

I think Wheel packages are the best option - they would create platform specific package with all files ready to be copied so there will be no need to build anything on target environments.

Answer 1

You can use auditwheel to inject the external libraries into the wheel:

auditwheel repair: copies these external shared libraries into the wheel itself, and automatically modifies the appropriate RPATH entries such that these libraries will be picked up at runtime. This accomplishes a similar result as if the libraries had been statically linked without requiring changes to the build system. Packagers are advised that bundling, like static linking, may implicate copyright concerns.

You can pre-build the external c++ library by typically executing the following:

./configure && make && make install

This will generate an my_external_library.so file and install it in the appropriate path. However, you'll need to ensure that the library path is properly set in order for the auditwheel to discover the missing dependency.

export LD_LIBRARY_PATH=/usr/loca/lib

You can then build the python wheel by executing:

python setup.py bdist_wheel

Finally, you can repair the wheel, which will inject the my_external_library.so into the package.

auditwheel repair my-python-wheel-1.5.2-cp35-cp35m-linux_x86_64.whl

I successfully applied the above steps to the python library confluent-kafka-python which has a required c/c++ dependency on librdkafka.

Answer 2

Wheels are the standard way of distributing Python packages, but there is a problem when you have extension modules that depend on other so's. This is because the normal Linux dynamic linker is used, and that only looks in /usr/lib or /usr/local/lib. This is a problem when installing a wheel in a virtualenv.

As far as I know, you have three options:

• Static linking, so the 'wrapper' does not depend on anything else;
• using ctypes to wrap the so directly from Python;
• Split the distribution into a wheel with the Python code & wrapper, and a separate RPM or DEB to install the so into either /usr/lib or /usr/local/lib.

A wheel may work if you include the dependent so as a data file to be stored in /lib and install into the root Python environment (haven't tried that), but this will break if someone tries to install the wheel into a virtualenv (did try that).