difficulties using setuptools to compile extension module

问题

I am developing a package (module?) designed to be pip-installed. The package contains a C++ extension, wrapped using pybind11, which links to some other libraries.

I am able to compile and link the extension libraries manually without issue, however I am unable to configure my setup.py file properly such that the extension module installs when invoking pip3 install my_package.

I have been following this code (which is linked to by the pybind11 documentation) to configure my setup.py file, which is included below.

Description of the Problem

For completeness, my package directory looks something like this:

my_package
    |-setup.py
    |-my_package
        |-__init__.py
        |-some_code.py
        |-src
            | my_extension.cpp
            | some_header.h

I invoke pip3 install as follows:

pip3 install --user --upgrade --disable-pip-version-check . -vvv

(the version check is disabled as the machine does not have access to the internet).

I can see from the output to the console that gcc appears to be invoked, but I do not know enough about the output to understand if it is invoked correctly. my_extension appears to be built without any errors:

building 'my_extension' extension
    creating build/temp.linux-x86_64-3.7
    creating build/temp.linux-x86_64-3.7/my_package
    creating build/temp.linux-x86_64-3.7/my_package/src
    gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/local/include/python3.7m -I/home/<my_user>/.local/include/python3.7m -I/usr/include/eigen3 -I/usr/include/python3.7m -c my_package/src/my_extension.cpp -o build/temp.linux-x86_64-3.7/my_package/src/my_extension.o -O3 -fPIC -march=native -DVERSION_INFO="0.0.1" -std=c++14 -fvisibility=hidden

However, for whatever reason, my_extension cannot be found by the python interpreter. Locating the src submodule:

>>> from my_package import src
>>> src.__path__
['/home/<my_user>/.local/lib/python3.7/site-packages/my_package/src']

I can see that it is empty:

>>> ls /home/<my_user>/.local/lib/python3.7/site-packages/my_package/src
__init__.py  __pycache__

that is, there does not appear to be a my_extension.so file here.

Questions

• Are there any obvious problems with the setup.py below? I appreciate this is a somewhat complicated example.

• If not, how might I go about troubleshooting this issue further? I am completely stuck with how to procede.

My setup.py

from setuptools import setup, Extension
from setuptools.command.build_ext import build_ext
import sys
import setuptools

__version__ = '0.0.1'

class get_pybind_include(object):

    def __init__(self, user=False):
        self.user = user

    def __str__(self):
        import pybind11
        return pybind11.get_include(self.user)

ext_modules = [
    Extension(
        'my_extension',
        ['my_package/src/my_extension.cpp'],
        include_dirs=[
            get_pybind_include(),
            get_pybind_include(user=True),
            '/usr/include/eigen3'
        ],
        language='c++'
    ),
]

def has_flag(compiler, flagname):
    import tempfile
    with tempfile.NamedTemporaryFile('w', suffix='.cpp') as f:
        f.write('int main (int argc, char **argv) { return 0; }')
        try:
            compiler.compile([f.name], extra_postargs=[flagname])
        except setuptools.distutils.errors.CompileError:
            return False
    return True


def cpp_flag(compiler):
    if has_flag(compiler, '-std=c++14'):
        return '-std=c++14'
    elif has_flag(compiler, '-std=c++11'):
        return '-std=c++11'
    else:
        raise RuntimeError('Unsupported compiler -- at least C++11 support is needed!')


class BuildExt(build_ext):
    """A custom build extension for adding compiler-specific options."""
    c_opts = {
        'msvc': ['/EHsc'],
        'unix': [],
    }

    if sys.platform == 'darwin':
        c_opts['unix'] += ['-stdlib=libc++', '-mmacosx-version-min=10.7']

    def build_extensions(self):
        ct = self.compiler.compiler_type
        opts = self.c_opts.get(ct, [])

        # extra compiler options
        opts += [
            '-O3',
            '-fPIC',
            '-march=native'
        ]

        if ct == 'unix':
            opts.append('-DVERSION_INFO="%s"' % self.distribution.get_version())
            opts.append(cpp_flag(self.compiler))
            if has_flag(self.compiler, '-fvisibility=hidden'):
                opts.append('-fvisibility=hidden')
        elif ct == 'msvc':
            opts.append('/DVERSION_INFO=\\"%s\\"' % self.distribution.get_version())
        for ext in self.extensions:
            ext.extra_compile_args = opts
        build_ext.build_extensions(self)

setup(
    name='my_package',
    version=__version__,
    author='',
    author_email='',
    url='',
    description='',
    long_description='',
    packages = setuptools.find_packages(),
    ext_modules=ext_modules,
    install_requires=['pybind11>=2.2'],
    cmdclass={'build_ext': BuildExt},
    zip_safe=False,
)

回答1:

Extension('my_extension', …) creates a top-level extension that you can import with

import my_extension

To make it a submodule of your package prepend the name of the package:

Extension('my_package.my_extension', …)

来源：https://stackoverflow.com/questions/58116773/difficulties-using-setuptools-to-compile-extension-module