Embed Python/C API in a multi-threading C++ program

问题

I am trying to embed Python in a C++ multi-threading program using the Python/C API (version 3.7.3) on a quad-core ARM 64 bit architecture. A dedicated thread-safe class "PyHandler" takes care of all the Python API calls:

class PyHandler
{
 public:
        PyHandler();
        ~PyHandler();
        bool run_fun();
        // ...
 private:
        PyGILState_STATE _gstate;
        std::mutex _mutex;
}

In the constructor I initialize the Python interpreter:

PyHandler::PyHandler()
{
    Py_Initialize();

    //PyEval_SaveThread(); // UNCOMMENT TO MAKE EVERYTHING WORK !
}

And in the destructor I undo all initializations:

PyHandler::~PyHandler()
{
    _gstate = PyGILState_Ensure();

    if (Py_IsInitialized()) // finalize python interpreter
        Py_Finalize();
}

Now, in order to make run_fun() callable by one thread at a time, I use the mutex variable _mutex (see below). On top of this, I call PyGILState_Ensure() to make sure the current thread holds the python GIL, and call PyGILState_Release() at the end to release it. All the remaining python calls happen within these two calls:

bool PyHandler::run_fun()
{
      std::lock_guard<std::mutex> lockGuard(_mutex);

      _gstate = PyGILState_Ensure(); // give the current thread the Python GIL

      // Python calls...

      PyGILState_Release(_gstate); // release the Python GIL till now assigned to the current thread

      return true;
}

Here is how the main() looks like:

int main()
{
      PyHandler py; // constructor is called !

      int n_threads = 10;
      std::vector<std::thread> threads;

      for (int i = 0; i < n_threads; i++)
            threads.push_back(std::thread([&py]() { py.run_fun(); }));

      for (int i = 0; i < n_threads; i++)
            if (threads[i].joinable())
                  threads[i].join();
}

Although all precautions, the program always deadlocks at the PyGILState_Ensure() line in run_fun() during the very first attempt. BUT when I uncomment the line with PyEval_SaveThread() in the constructor everything magically works. Why is that ?

Notice that I am not calling PyEval_RestoreThread() anywhere. Am I supposed to use the macros Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS instead ? I thought these macros and PyEval_SaveThread() are only used dealing with Python threads and NOT with non-Python threads, as in my case! Am I missing something ?

The documentation for my case, only mentions the use of PyGILState_Ensure() and PyGILState_Release. Any help is highly appreciated.

来源：https://stackoverflow.com/questions/64429383/embed-python-c-api-in-a-multi-threading-c-program

标签

python

c++

multithreading