Distutils: build multiple Python extension modules (written in Swig) that share a method

问题

I have four C++ files: A.h, A.cpp, B.h, B.cpp, and A.h is included in B.cpp

A.h:

#pragma once 
void A();

A.cpp:

#include <iostream>
void A() {
    std::cout << "A" << std::endl;
}

B.h:

#pragma once
void B();

B.cpp:

#include "A.h"
#include <iostream>
void B() {
    A();
    std::cout << "B" << std::endl;
}

Now I wrote two SWIG inerface files A.i and B.i

A.i:

%module A
%{
#include "A.h"
%}
%include "A.h"

B.i:

%module B
%{
#include "B.h"
%}
%include "B.h"

The setup.py file is:

from distutils.core import setup, Extension
A_ext = Extension( "_A", [ "A.i", "A.cpp", ], swig_opts = ['-c++'], extra_compile_args = ['-g'])
B_ext = Extension( "_B", [ "B.i", "B.cpp", ], swig_opts = ['-c++'], extra_compile_args = ['-g'])
setup(
    name = "test",
    version = "1.0",
    ext_modules = [ A_ext, B_ext ],
    py_modules = [ "A", "B" ]
)

If I type the command below, it will show 'A'.

python -c 'import A; A.A()'

If I type the command below, the segmentation fault appears:

python -c 'import B; B.B()'

How could I do to get this command run correctly? Since I don't want to compile B.cpp many times, is there any way except the one below?

B_ext = Extension( "_B", [ "B.i", "A.cpp", "B.cpp", ], swig_opts = ['-c++'], extra_compile_args = ['-g'])

回答1:

I changed your files a bit, for clarity.

a.h:

#pragma once 


void funcA();

a.cpp:

#include <iostream>


void funcA() {
    std::cout << __FILE__ << " " << __LINE__ <<  " " << __FUNCTION__ <<  std::endl;
}

a.i:

%module a
%{
    #include "a.h"
%}
%include "a.h"

b.h:

#pragma once


void funcB();

b.cpp:

#include "a.h"
#include <iostream>


void funcB() {
    std::cout << __FILE__ << " " << __LINE__ <<  " " << __FUNCTION__ <<  std::endl;
    funcA();
}

b.i:

%module b
%{
    #include "b.h"
%}
%include "b.h"

setup.py:

from distutils.core import setup
from distutils.extension import Extension


a = "a"
b = "b"

ext_a = Extension("_" + a, [a + ".i", a + ".cpp"], swig_opts=("-c++",), extra_compile_args=["-g"])
ext_b = Extension("_" + b, [b + ".i", b + ".cpp"], swig_opts=("-c++",), extra_compile_args=["-g"])

setup(
    name="test",
    version="1.0",
    ext_modules=[ext_a, ext_b],
    py_modules=[a, b]
)

What happens (simplified) when calling b.funcB() (only the stacktrace, imports left aside). Each step invokes the next:

funcB from module b (b.py)
funcB from module _b (_b.so, or _b.cpython-35m-x86_64-linux-gnu.so)
- Everything from here happens in C (or C++)
- Current funcB is different than the one from b.cpp: it's generated by swig and its name is _wrap_funcB
- Previous bullet also applies to funcA and a.cpp
funcB from b.cpp
funcA from a.cpp

The problem is that the code from step #4. is not in the module _b, and it will fail at runtime. But things are a little bit stranger: the failure (core dump) doesn't appear when funcB is called but at module (b -> _b) import time (I think this happens because of swig's behind the scene magic), as seen below.

Output:

[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q050938128]> ls
a.cpp  a.h  a.i  b.cpp  b.h  b.i  setup.py
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q050938128]> python3 setup.py build > /dev/null 2>&1
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q050938128]> echo $?
0
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q050938128]> ls
a.cpp  a.h  a.i  a.py  a_wrap.cpp  b.cpp  b.h  b.i  b.py  build  b_wrap.cpp  setup.py
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q050938128]> ls ./build/lib.linux-x86_64-3.5
_a.cpython-35m-x86_64-linux-gnu.so  _b.cpython-35m-x86_64-linux-gnu.so
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q050938128]> PYTHONPATH=${PYTHONPATH}:./build/lib.linux-x86_64-3.5 python3 -c "import _a;_a.funcA()"
a.cpp 6 funcA
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q050938128]> PYTHONPATH=${PYTHONPATH}:./build/lib.linux-x86_64-3.5 python3 -c "import _b"
Segmentation fault (core dumped)

In order to solve it, either:

As you pointed out include funcA in module _b (by adding a.cpp in ext_b's source files list). This way, both modules will be self contained (from funcA's PoV), each will work independently, but also funcA will be duplicated in both of them
Make _b depend on _a (after all, they're shared objects). But that's not how Python extension modules are to be used, and it won't work on Win (and also on some Ux flavors). So, this is more like a (lame) workaround (gainarie)
Build a.cpp into a different shared library (an .so, but not a Python extension module) to be used by both modules. Needless to say that at runtime each of them will require the .so to be present

Obviously, to option #3. is the perfect candidate. But distutils ([Python 3]: API Reference) doesn't provide the needed functionality OOTB (apparently building an extension module and an external shared library that it depends on, is not a scenario that distutils aims for), or at least, I couldn't find any.
There is a build_clib module, which offers the functionality of building a static lib (to be used by the extension modules), but that would be same as option #1..

setup.py:

import sys
import os
from distutils.core import setup
from distutils.extension import Extension
from distutils.command.build_clib import build_clib
from distutils.command.build_ext import build_ext
from distutils.ccompiler import CCompiler


__win = sys.platform[:3].lower() == "win"
export_symbols_option = "export_symbols"


class build_clib_dyn(build_clib):
    def finalize_options(self):
        self.set_undefined_options('build',
                                   ('build_lib', 'build_clib'),
                                   ('build_temp', 'build_temp'),
                                   ('compiler', 'compiler'),
                                   ('debug', 'debug'),
                                   ('force', 'force'))
        self.libraries = self.distribution.libraries
        if self.libraries:
            self.check_library_list(self.libraries)
        if self.include_dirs is None:
            self.include_dirs = self.distribution.include_dirs or []
        if isinstance(self.include_dirs, str):
            self.include_dirs = self.include_dirs.split(os.pathsep)

    def build_libraries(self, libraries):
        for (lib_name, build_info) in libraries:
            sources = build_info.get('sources')
            if sources is None or not isinstance(sources, (list, tuple)):
                raise DistutilsSetupError(
                       "in 'libraries' option (library '%s'), "
                       "'sources' must be present and must be "
                       "a list of source filenames" % lib_name)
            sources = list(sources)
            macros = build_info.get('macros')
            include_dirs = build_info.get('include_dirs')
            objects = self.compiler.compile(sources,
                                            output_dir=self.build_temp,
                                            macros=macros,
                                            include_dirs=include_dirs,
                                            debug=self.debug)
            self.compiler.link(CCompiler.SHARED_OBJECT, objects, self.compiler.library_filename(lib_name, lib_type="shared"),
                               output_dir=self.build_clib,
                               export_symbols=build_info.get(export_symbols_option),
                               debug=self.debug)


if __win:
    class build_ext_w_dyn_dep(build_ext):
        def finalize_options(self):
            super(build_ext_w_dyn_dep, self).finalize_options()
            self.library_dirs.append(os.path.dirname(self.build_temp))

else:
    class build_ext_w_dyn_dep(build_ext):
        pass


a_name = "a"
b_name = "b"
common_name = a_name + b_name + "common"
swig_opts = ["-c++"]
libraries = [common_name]
lib_common_build_info = {"sources": [a_name + ".cpp"]}
if __win:
    extra_compile_args = None
    extra_link_args = None
    lib_common_build_info[export_symbols_option] = ["funcA"]
else:
    extra_compile_args = ["-g"]
    extra_link_args = ["-Wl,-rpath,${ORIGIN}"]

lib_common_info = (common_name, lib_common_build_info)
ext_a = Extension("_" + a_name, [a_name + ".i"], libraries=libraries, extra_compile_args=extra_compile_args, extra_link_args=extra_link_args, swig_opts=swig_opts)
ext_b = Extension("_" + b_name, [b_name + ".i", b_name + ".cpp"], libraries=libraries, extra_compile_args=extra_compile_args, extra_link_args=extra_link_args, swig_opts=swig_opts)

setup(
    name="test",
    version="1.0",
    libraries=[lib_common_info],
    cmdclass={"build_clib": build_clib_dyn, "build_ext": build_ext_w_dyn_dep},
    ext_modules=[ext_a, ext_b],
    py_modules=[a_name, b_name]
)

Notes:

build_clib_dyn extends build_clib as its functionality had to be modified. 2 methods overridden but only small parts of them actually changed (comments were not copied from the base class methods (Python3.5.4 codebase), in order to reduce the amount of code, but that doesn't really count as a change)
The task took quite some distutils code browsing, as some options are not documented (and I'm not very familiar with it)
Some Nix knowledge was required as well, as things get tricky when loading shared libraries that are not in the system paths, and also keeping things smooth (e.g. not altering ${LD_LIBRARY_PATH})
build_ext_w_dyn_dep is similar to build_clib_dyn (Win only). Since when building a Win dynamic link library (.dll), 2 files are generated, and in this case they are not in the same dir, library search paths needed some adjustments
Python 3 and Python 2 compatible

Output (running the above commands again):

[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q050938128]> ls
a.cpp  a.h  a.i  b.cpp  b.h  b.i  setup.py
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q050938128]> python3 setup.py build > /dev/null 2>&1
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q050938128]> echo $?
0
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q050938128]> ls
a.cpp  a.h  a.i  a.py  a_wrap.cpp  b.cpp  b.h  b.i  b.py  build  b_wrap.cpp  setup.py
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q050938128]> ls build/lib.linux-x86_64-3.5/
_a.cpython-35m-x86_64-linux-gnu.so  _b.cpython-35m-x86_64-linux-gnu.so  libabcommon.so
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q050938128]> ldd build/lib.linux-x86_64-3.5/_a.cpython-35m-x86_64-linux-gnu.so
        linux-vdso.so.1 =>  (0x00007fffadb49000)
        libabcommon.so => /home/cfati/Work/Dev/StackOverflow/q050938128/build/lib.linux-x86_64-3.5/libabcommon.so (0x00007f91cd50f000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f91cd18d000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f91ccf77000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f91ccbad000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f91cc990000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f91cc687000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f91cd916000)
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q050938128]> PYTHONPATH=${PYTHONPATH}:./build/lib.linux-x86_64-3.5 python3 -c "import _a;_a.funcA()"
a.cpp 6 funcA
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q050938128]> PYTHONPATH=${PYTHONPATH}:./build/lib.linux-x86_64-3.5 python3 -c "import _b;_b.funcB()"
b.cpp 7 funcB
a.cpp 6 funcA

@EDIT0:

Added support for Win (I guess, it's not very important for the question)
- Since the module structure is simple, it was possible to do everything at setup.py's level (didn't have to modify the source files)
- The only extra requirement is that swig.exe's dir should be in %PATH%

来源：https://stackoverflow.com/questions/50938128/distutils-build-multiple-python-extension-modules-written-in-swig-that-share

标签

python

c++

swig

distutils