Embedded Python: Multiple Sub-Interpreters not working

问题

I'm trying to understand sub-interpreters and GIL. But my experiment is failing often(The same code rarely works).

Gathering info from SO questions and few sites, I have the following code which spawns 2 non-python threads. Each of these threads are given a python sub-interpreter. I want to release GIL inside these threads and call a DLL function in C++(though this example does not detail that. Here I just write to stdout). Basically I want to see concurrency in the execution(Non-Python DLL invocation).

#include <iostream>
#include <thread>
#include <Python.h>

void worker(PyInterpreterState* interp, int n) 
{
    PyThreadState* ts;
    ts = PyThreadState_New(interp);

    PyThreadState_Swap(ts);

    PyThreadState* _save;
    _save = PyEval_SaveThread();

    std::cout << n << std::endl;  // Non-Python execution. My Focus.

    PyEval_RestoreThread(_save);

    PyThreadState_Swap(ts);

    PyThreadState_Clear(ts);
    PyThreadState_DeleteCurrent();

    return;
}


int main()
{
    try 
    {
        Py_Initialize();

        PyEval_InitThreads();

        PyThreadState* _main = PyThreadState_Get();

        PyThreadState* i1 = Py_NewInterpreter();
        PyThreadState* i2 = Py_NewInterpreter();

        std::thread t1(worker, i1->interp, 1);
        std::thread t2(worker, i2->interp, 2);

        t1.join();
        t2.join();

        PyThreadState_Swap(i1);
        PyThreadState_Clear(i1);
        Py_EndInterpreter(i1);
        PyThreadState_Swap(i2);
        PyThreadState_Clear(i2);
        Py_EndInterpreter(i2);

        PyThreadState_Swap(_main);

        Py_Finalize();
        return 0;
    }
    catch(std::exception& e)
    {
        std::cout << "Exception:" << std::endl << e.what() << std::endl;
    }
}

Running a single thread works all the time. When I run 2 threads as shown, I get any of the following outputs.

In PyEval_SaveThread(),

2
Fatal Python error: drop_gil: GIL is not locked
Python runtime state: initialized

Current thread 0x00002d08 (most recent call first):
<no Python frame>

In PyEval_SaveThread(),

1
Fatal Python error: PyEval_SaveThread: NULL tstate
Python runtime state: initialized

Current thread 0x00003eb8 (most recent call first):
<no Python frame>

Either of the thread succeeds, the other one fails.

Rarely works. The same code.

1
2

Can someone shed some light on this? Need help. Thanks.

回答1:

In your worker() function, you call PyThreadState_Swap(). The Python docs say that:

The global interpreter lock must be held and is not released.

You must acquire the GIL before calling PyThreadState_Swap() and release it before exiting worker().

In your main thread, you wait for the threads to terminate while holding the GIL. This is a deadlock as the threads will need to acquire the GIL to do any useful work in Python.

Please see my answer at https://stackoverflow.com/a/26570708/99279 for detailed working instructions on how to do that, and even a link to sample code.

Once you have the GIL, and the thread state for for the sub interpreter, so that it is now safe to call the regular Python API, you can just add a PyEval_SaveThread()/PyEval_RestoreThread() pair.

来源：https://stackoverflow.com/questions/59666352/embedded-python-multiple-sub-interpreters-not-working

标签

python

multithreading

python-c-api

python-embedding