PyGILState_Ensure() Causing Deadlock

Question

I\'m writing a Python extension in C++, wrapping a third-party library I do not control. That library creates a thread Python knows nothing about, and from that thread, calls a

Answer 1

I'm new to StackOverflow, but I've been working on embedding python in a multithreaded C++ system for the last few days and run into a fair number of situations where the code has deadlocked itself. Here's the solution that I've been using to ensure thread safety:

class PyContextManager {
   private:
      static volatile bool python_threads_initialized;
   public:
      static std::mutex pyContextLock;
      PyContextManager(/* if python_threads_initialized is false, call PyEval_InitThreads and set the variable to true */);
      ~PyContextManager();
};

#define PY_SAFE_CONTEXT(expr)                   \
{                                               \
   std::unique_lock<std::mutex>(pyContextLock); \
   PyGILState_STATE gstate;                     \
   gstate = PyGILState_Ensure();                \
      expr;                                     \
   PyGILState_Release(gstate);                  \
}

Initializing the boolean and the mutex in the .cpp file.

I've noticed that without the mutex, the PyGILState_Ensure() command can cause a thread to deadlock. Likewise, calling PySafeContext within the expr of another PySafeContext will cause the thread to brick while it waits on its mutex.

Using these functions, I believe your callback function would look like this:

void Wrapper::myCallback()
{
   PyContextManager cm();
   PY_SAFE_CONTEXT(
       PyObject *result=PyObject_CallMethod(_pyObj,"callback",nullptr);
       if (result) Py_DECREF(result);
   );
}

If you don't believe that your code is likely to ever need more than one multithreaded call to Python, you can easily expand the macro and take the static variables out of a class structure. This is just how I've handled an unknown thread starting and determining whether it needs to start up the system, and dodging the tedium of writing out the GIL functions repeatedly.

Hope this helps!

Answer 2

I have wrapped C++ observers in Python. If you are using boost then you can call PyEval_InitThreads() in BOOST_PYTHON_MODULE:

BOOST_PYTHON_MODULE(eapipy)
{
     boost::shared_ptr<Python::InitialisePythonGIL> gil(new Python::InitialisePythonGIL());
....
}

Then I use a class to control calling back into Python from C++.

struct PyLockGIL
{

    PyLockGIL()
        : gstate(PyGILState_Ensure())
    { 
    }

    ~PyLockGIL()
    {
        PyGILState_Release(gstate);
    }

    PyLockGIL(const PyLockGIL&) = delete;
    PyLockGIL& operator=(const PyLockGIL&) = delete;

    PyGILState_STATE gstate;
};

If you are calling into C++ for any length of time you can also relinquish the GIL:

struct PyRelinquishGIL
{
    PyRelinquishGIL()
        : _thread_state(PyEval_SaveThread())
    {
    }
    ~PyRelinquishGIL()
    {
        PyEval_RestoreThread(_thread_state);
    }

    PyRelinquishGIL(const PyLockGIL&) = delete;
    PyRelinquishGIL& operator=(const PyLockGIL&) = delete;

    PyThreadState* _thread_state;
};

Our code is multi-threaded and this approach works well.

Answer 3

This answer is only for Python >= 3.0.0. I don't know if it would work for earlier Pythons or not.

Wrap your C++ module in a Python module that looks something like this:

import threading
t = threading.Thread(target=lambda: None, daemon=True)
t.run()
del t
from your_cpp_module import *

From my reading of the documentation, that should force threading to be initialized before your module is imported. Then the callback function you have written up there should work.

I'm less confident of this working, but your module init function could instead do this:

if (!PyEval_ThreadsInitialized())
{
    PyEval_InitThreads();
}

that should work because your module init function should be being executed by the only Python thread in existence if PyEval_ThreadsInitialized() isn't true, and holding the GIL is the right thing to do then.

These are guesses on my part. I've never done anything like this as is evidenced by my clueless comments on your question. But from my reading of the documentation, both of these approaches should work.