How to compile __init__.py file using cython on Windows

Question

When I compile an arbitrary __init__.py file on Windows with setup.py build_ext --inplace command, it has an unresolvable external symbol error (i.e. "LINK

Answer 1

Maybe this behaviour might be viewed as a small bug in distutils-package (as pointed out by @DavidW there is this open issue: https://bugs.python.org/issue35893). However, it also shows, that cythonizing/compiling __init__.py isn't very popular and uses some undocumented implementation details which might change in the future, so it could be wiser to refrain from meddling with __init__.py.

But if you must...

---

When a package is imported explicitly, e.g.

import ctest

or implicitly, e.g.

import ctest.something

The FileFinder will see that a package, and not a module, is imported and will try to load ctest/__init__.py instead of ctest.py (which most likely doesn't exists):

    # Check if the module is the name of a directory (and thus a package).
    if cache_module in cache:
        base_path = _path_join(self.path, tail_module)
        for suffix, loader_class in self._loaders:
            init_filename = '__init__' + suffix
            full_path = _path_join(base_path, init_filename)
            if _path_isfile(full_path):
                return self._get_spec(loader_class, fullname, full_path, [base_path], target)

Used suffix, loader_class are for loading __init__.so, __init__.py and __init__.pyc in this order (see also this SO-post). This means, __init__.so will be loaded instead of __init__.py if we manage to create one.

While __init__.py is executed, The property __name__ is the name of the package, i.e. ctest in your case, and not __init__ as one might think. Thus, the name of the init-function, Python-interpreter will call when loading the extension __init__.so is PyInit_ctest in your case (and not PyInit___init__ as one might think).

The above explains, why it all works on Linux out-of-the-box. What about Windows?

The loader can only use symbols from a so/dll which aren't hidden. Per default all symbols built with gcc are visible, but not for VisualStudio on Windows - where all symbols are hidden per default (see e.g. this SO-post).

However, the init-function of a C-extension must be visible (and only the init-function) so it can be called with help of the loader - the solution is to export this symbol (i.e. PyInit_ctest) while linking, in your case it is the wrong /EXPORT:PyInit___init__-option for the linker.

The problem can be found in distutils, or more precise in build_ext-class:

def get_export_symbols(self, ext):
    """Return the list of symbols that a shared extension has to
    export.  This either uses 'ext.export_symbols' or, if it's not
    provided, "PyInit_" + module_name.  Only relevant on Windows, where
    the .pyd file (DLL) must export the module "PyInit_" function.
    """
    initfunc_name = "PyInit_" + ext.name.split('.')[-1]
    if initfunc_name not in ext.export_symbols:
        ext.export_symbols.append(initfunc_name)
    return ext.export_symbols

Here, sadly ext.name has __init__ in it.

From here, one possible solution is easy : to override get_export_symbols, i.e. to add the following to your setup.py-file (read on for a even simpler version):

...
from distutils.command.build_ext import build_ext
def get_export_symbols_fixed(self, ext):
    names = ext.name.split('.')
    if names[-1] != "__init__":
        initfunc_name = "PyInit_" + names[-1]
    else:
        # take name of the package if it is an __init__-file
        initfunc_name = "PyInit_" + names[-2]
    if initfunc_name not in ext.export_symbols:
        ext.export_symbols.append(initfunc_name)
    return ext.export_symbols

# replace wrong version with the fixed:
build_ext.get_export_symbols = get_export_symbols_fixed
...

Calling python setup.py build_ext -i should be enough now (because __init__.so will be loaded rather than __init__.py).

---

However, as @DawidW has pointed out, Cython uses macro PyMODINIT_FUNC, which is defined as

#define PyMODINIT_FUNC Py_EXPORTED_SYMBOL PyObject*

with Py_EXPORTED_SYMBOL being marked as visible/exported on Windows:

#define Py_EXPORTED_SYMBOL __declspec(dllexport)

Thus, there is no need to mark the symbol as visible at the command line. Even worse, this is the reason for the warning LNK4197:

__init__.obj : warning LNK4197: export 'PyInit_ctest' specified multiple times; using first specification

as PyInit_test is marked as __declspec(dllexport) and exported via option /EXPORT: at the same time.

/EXPORT:-option will be skipped by distutils, if export_symbols is empty, we can use even a simpler version of command.build_ext:

...
from distutils.command.build_ext import build_ext
def get_export_symbols_fixed(self, ext):
    pass  # return [] also does the job!

# replace wrong version with the fixed:
build_ext.get_export_symbols = get_export_symbols_fixed
...

This is even better than the first version, as it also fixes warning LNK4197!

Answer 2

This is a very tentative answer because I have no easy way of testing it on Windows, so if it's wrong then let me know and I'll delete it.

Can you try running (on the commend line):

C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:d:\py37\Libs /LIBPATH:D:\ENVS\cpytrantest\libs /LIBPATH:D:\ENVS\cpytrantest\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" /EXPORT:PyInit_ctest build\temp.win-amd64-3.7\Release\ctest/__init__.obj /OUT:C:\Users\76923\Desktop\cpythonrecord\ctest\__init__.cp37-win_amd64.pyd /IMPLIB:build\temp.win-md64-3.7\Release\ctest\__init__.cp37-win_amd64.lib

All I've done is taken the compilation command that distutils generated and replaced /EXPORT:PyInit___init__ with /EXPORT:PyInit_ctest. The /EXPORT is a Windows specific compiler option that doesn't get added on Linux. It looks like either distutils or Cython passes the name PyInit___init__ to MSVC, but if I look in the actual generated C file then the name appears to be PyInit_ctest, hence the undefined symbol.

If that workaround (doing the compilation independently of distutils) works then you should report the bug to either the distutils or the Cython bug tracker (probably Cython) with these details and hopefully it can be fixed.