Why is PyGILState_Release throwing Fatal Python Errors

Question

ANSWERED

Ok, I solved this issue. Its all in how you initialize the thread state. You don't need to use ReleaseLock at all. Simply add InitThreads call to your module definition:

BOOST_PYTHON_MODULE(ModuleName)
{
    PyEval_InitThreads();

    ...
}

---

Ok, I have attempted to diagnose this problem for hours and poured through what seems like every example on the web. Getting tired now so I may be missing something obvious but here is what is happening:

I am wrapping a library in boost python. I am running a python script which imports the lib, constructs some objects and then receives callbacks from c++ which calls back into python. Before I invoke call any python functions, I attempt to acquire the global interpreter lock. Here is some sample code:

class ScopedGILRelease
{
public:
   inline ScopedGILRelease()
   {
      d_gstate = PyGILState_Ensure();
   }

   inline ~ScopedGILRelease()
   {
      PyGILState_Release(d_gstate);
   }

private:
   PyGILState_STATE  d_gstate;
};

class PyTarget : public DingoClient::ClientRequest::Target, public wrapper<DingoClient::ClientRequest::Target>
{
  public:
    PyTarget(PyObject* self_) : self(self_) {}
    ~PyTarget() {
      ScopedGILRelease gil_lock;
    }
    PyObject* self;

    void onData(const boost::shared_ptr<Datum>::P & data, const void * closure)
    {
       ScopedGILRelease gil_lock;
       // invoke call_method to python 
    }

    ...
}

The onData method on the Target object is called by the library as a callback. In python, we inherit from PyTarget and implement another method. We then use call_method<> to call that method. gil_lock acquires the lock and through RIAA guarantees that the thread state acquired is always the one release and that it is in fact always released when going out of scope.

However when I run this in a script which attempts to get a large number of callbacks on this function, it always segfaults. Script looks something like this:

# Initialize the library and setup callbacks
...

# Wait until user breaks
while 1:
  pass

Also, the python script always constructs an object which runs:

PyEval_InitThreads();
PyEval_ReleaseLock();

before receiving any callbacks.

I've reduced the code to where I'm not even calling into python in onData, I'm just acquiring the lock. On release it always crashes with either:

Fatal Python error: ceval: tstate mix-up
Fatal Python error: This thread state must be current when releasing

or

Fatal Python error: ceval: orphan tstate
Fatal Python error: This thread state must be current when releasing

It is seemingly random. Am I crazy here, because I feel like I'm using the GIL lock correctly, however it doesn't seem to work at all.

Other Notes: Only one thread ever calls that Target object's onData method.

When I sleep in the while loop in the calling python module with time.sleep(), it seems to allow the script to run longer but eventually script will segfault with similar problems. The amount of time it lasts is proportional to the amount of time.sleep (i.e. time.sleep(10) runs longer than time.sleep(0.01). This makes me think some how the script is re-acquiring the GIL without my permission.

PyGILState_Release and PyGILState_Ensure is called no where else in my code, no where else should be calling into python.

Update

I've read another question which suggests import threading in the module as an alternative to running

PyEval_InitThreads();
PyEval_ReleaseLock();

However, it does not appear to work when I import threading before my module and remove the above two lines from my boost python wrapper.

Answer 1

Ok, I solved this issue. Its all in how you initialize the thread state. You don't need to use ReleaseLock at all. Simply add InitThreads call to your module definition:

BOOST_PYTHON_MODULE(ModuleName)
{
    PyEval_InitThreads();

    ...
}