PyGILState_Ensure() Causing Deadlock

Question

I\'m writing a Python extension in C++, wrapping a third-party library I do not control. That library creates a thread Python knows nothing about, and from that thread, calls a

Answer 1

I'm new to StackOverflow, but I've been working on embedding python in a multithreaded C++ system for the last few days and run into a fair number of situations where the code has deadlocked itself. Here's the solution that I've been using to ensure thread safety:

class PyContextManager {
   private:
      static volatile bool python_threads_initialized;
   public:
      static std::mutex pyContextLock;
      PyContextManager(/* if python_threads_initialized is false, call PyEval_InitThreads and set the variable to true */);
      ~PyContextManager();
};

#define PY_SAFE_CONTEXT(expr)                   \
{                                               \
   std::unique_lock(pyContextLock); \
   PyGILState_STATE gstate;                     \
   gstate = PyGILState_Ensure();                \
      expr;                                     \
   PyGILState_Release(gstate);                  \
}

Initializing the boolean and the mutex in the .cpp file.

I've noticed that without the mutex, the PyGILState_Ensure() command can cause a thread to deadlock. Likewise, calling PySafeContext within the expr of another PySafeContext will cause the thread to brick while it waits on its mutex.

Using these functions, I believe your callback function would look like this:

void Wrapper::myCallback()
{
   PyContextManager cm();
   PY_SAFE_CONTEXT(
       PyObject *result=PyObject_CallMethod(_pyObj,"callback",nullptr);
       if (result) Py_DECREF(result);
   );
}

If you don't believe that your code is likely to ever need more than one multithreaded call to Python, you can easily expand the macro and take the static variables out of a class structure. This is just how I've handled an unknown thread starting and determining whether it needs to start up the system, and dodging the tedium of writing out the GIL functions repeatedly.

Hope this helps!