Embedding python in multithreaded C application

Question

I\'m embedding the python interpreter in a multithreaded C application and I\'m a little confused as to what APIs I should use to ensure thread safety.

From what I g

Answer 1

I had exactly the same problem and it is now solved by using PyEval_SaveThread() immediately after PyEval_InitThreads(), as you suggest above. However, my actual problem was that I used PyEval_InitThreads() after PyInitialise() which then caused PyGILState_Ensure() to block when called from different, subsequent native threads. In summary, this is what I do now:

1. There is global variable:

   static int gil_init = 0; 
   

2. From a main thread load the native C extension and start the Python interpreter:

   Py_Initialize() 
   

3. From multiple other threads my app concurrently makes a lot of calls into the Python/C API:

   if (!gil_init) {
       gil_init = 1;
       PyEval_InitThreads();
       PyEval_SaveThread();
   }
   state = PyGILState_Ensure();
   // Call Python/C API functions...    
   PyGILState_Release(state);
   

4. From the main thread stop the Python interpreter

   Py_Finalize()
   

All other solutions I've tried either caused random Python sigfaults or deadlock/blocking using PyGILState_Ensure().

The Python documentation really should be more clear on this and at least provide an example for both the embedding and extension use cases.

Answer 2

Having a multi-threaded C app trying to communicate from multiple threads to multiple Python threads of a single CPython instance looks risky to me.

As long as only one C thread communicates with Python you should not have to worry about locking even if the Python application is multi-threading. If you need multiple python threads you can set the application up this way and have multiple C threads communicate via a queue with that single C thread that farms them out to multiple Python threads.

An alternative that might work for you is to have multiple CPython instances one for each C thread that needs it (of course communication between Python programs should be via the C program).

Another alternative might the Stackless Python interpreter. That does away with the GIL, but I am not sure you run into other problems binding it to multiple threads. stackless was a drop-in replacement for my (single-threaded) C application.

Answer 3

Eventually I figured it out.
After

PyEval_InitThreads();

You need to call

PyEval_SaveThread();

While properly release the GIL for the main thread.