Pybind11 multiprocessing hangs

Question

I\'m writing an application that use Pybind11 to embed Python interpreter (Windows, 64 bit, Visual C++ 2017). From Python, I need to spawn multiple processes, but it seems d

Answer 1

Ok, thanks to this link: https://blender.stackexchange.com/questions/8530/how-to-get-python-multiprocessing-module-working-on-windows, I solved this strange issue (that seems to be Windows related). It's not a Pybind11 issue, but a Python C API itself. You can solve the issue by setting sys.executable equals to the path of the python interpreter executable (python.exe) and by writing the python code to a file and setting the path to the __file__ variable. That is, I have to add:

import sys
sys.executable = "C:\\Users\\MyUserName\\Miniconda3\\python.exe"
__file__ = "C:\\tmp\\run.py"

and I need to write the python code to the file specified by __file__, that is:

FILE *f = nullptr;
fopen_s(&f, "c:\\tmp\\run.py", "wt");
fprintf(f, "%s", pyCode.c_str());
fclose(f);

just before execute py::exec(pyCode).

In addition the code:

if len(sys.argv) > 1:
    __name__ = "__mp_main__"

is no longer necessary. However, note that in this way the runned processes are not embedded anymore and, unfortunately, if you want to directly pass a C++ module to them, you cannot do it.

Hope this can help someone else.