Embedding python in multithreaded C application

Question

I'm embedding the python interpreter in a multithreaded C application and I'm a little confused as to what APIs I should use to ensure thread safety.

From what I gathered, when embedding python it is up to the embedder to take care of the GIL lock before calling any other Python C API call. This is done with these functions:

gstate = PyGILState_Ensure();
// do some python api calls, run python scripts
PyGILState_Release(gstate);

But this alone doesn't seem to be enough. I still got random crashes since it doesn't seem to provide mutual exclusion for the Python APIs.

After reading some more docs I also added:

PyEval_InitThreads();

right after the call to Py_IsInitialized() but that's where the confusing part comes. The docs state that this function:

Initialize and acquire the global interpreter lock

This suggests that when this function returns, the GIL is supposed to be locked and should be unlocked somehow. but in practice this doesn't seem to be required. With this line in place my multithreaded worked perfectly and mutual exclusion was maintained by the PyGILState_Ensure/Release functions.
When I tried adding PyEval_ReleaseLock() after PyEval_ReleaseLock() the app dead-locked pretty quickly in a subsequent call to PyImport_ExecCodeModule().

So what am I missing here?

Answer 1

I had exactly the same problem and it is now solved by using PyEval_SaveThread() immediately after PyEval_InitThreads(), as you suggest above. However, my actual problem was that I used PyEval_InitThreads() after PyInitialise() which then caused PyGILState_Ensure() to block when called from different, subsequent native threads. In summary, this is what I do now:

1. There is global variable:

   static int gil_init = 0; 
   

2. From a main thread load the native C extension and start the Python interpreter:

   Py_Initialize() 
   

3. From multiple other threads my app concurrently makes a lot of calls into the Python/C API:

   if (!gil_init) {
       gil_init = 1;
       PyEval_InitThreads();
       PyEval_SaveThread();
   }
   state = PyGILState_Ensure();
   // Call Python/C API functions...    
   PyGILState_Release(state);
   

4. From the main thread stop the Python interpreter

   Py_Finalize()
   

All other solutions I've tried either caused random Python sigfaults or deadlock/blocking using PyGILState_Ensure().

The Python documentation really should be more clear on this and at least provide an example for both the embedding and extension use cases.

Answer 2

Eventually I figured it out.
After

PyEval_InitThreads();

You need to call

PyEval_SaveThread();

While properly release the GIL for the main thread.

Answer 3

Having a multi-threaded C app trying to communicate from multiple threads to multiple Python threads of a single CPython instance looks risky to me.

As long as only one C thread communicates with Python you should not have to worry about locking even if the Python application is multi-threading. If you need multiple python threads you can set the application up this way and have multiple C threads communicate via a queue with that single C thread that farms them out to multiple Python threads.

An alternative that might work for you is to have multiple CPython instances one for each C thread that needs it (of course communication between Python programs should be via the C program).

Another alternative might the Stackless Python interpreter. That does away with the GIL, but I am not sure you run into other problems binding it to multiple threads. stackless was a drop-in replacement for my (single-threaded) C application.