Limiting execution time of embedded Python

Question

If I embed the Python interpreter in a C or C++ program, as in this example, is there any way to limit how long the interpreter runs for? Is there anything to stop the Python code from entering an infinite loop and thus preventing PyObject_CallObject (or equivalent) from ever returning?

Similarly, if the Python code creates a new thread, is there anything to stop this thread from entering an infinite loop and running forever?

Answer 1

As you can see in the docs, PyObject_CallObject has no mechanism for limiting how long the function runs. There is also no Python C API function that I am aware of that allows you to pause or kill a thread used by the interpreter.

We therefore have to be a little more creative in how we stop a thread running. I can think of 3 ways you could do it (from safest/cleanest to most dangerous...

Poll your main application

The idea here is that your Python function which could run for a long time simply calls another function inside your main application, using the C API to see if it should shut down. A simple True/False result would allow you to terminate gracefully.

This is the safest solution, but requires that you alter your Python code.

Use exceptions

Since you are embedding the Interpreter, you are already using the C API and so could use PyThreadState_SetAsyncExc to force an exception to be raised in the offending thread. You can find an example that uses this API here. While it is Python code, the same function will work from your main application.

This solution is a little less safe as it requires the code not to catch the Exception and to remain in a usable state afterwards.

Use the OS to terminate the thread

I'm not going to go into this one as it is inherently unsafe. See Is there any way to kill a Thread in Python? for some explanations of why.

Answer 2

If You need control lag and jitter in some realtime system, embed "StacklessPython" - Python with support co-routines. Used in "Twisted Web Server".

Answer 3

I wanted to be able to interrupt any actively running Python code block, including cancelling an infinite loop or just some long-running code. In my application, single-threaded Python code is submitted and evaluated. An interrupt request can come at any time. This question and answer were crucial in helping me actually achieve that.

I use the middle approach from @Peter Brittain's answer from July 2016 above.

After I start the Python engine, I retrieve the thread id using the threading package in Python code. I parse that into a c long value and save it.

The interrupt function is actually quite simple:
PyGILState_Ensure()
PyThreadState_SetAsyncExc(threadId, PyExc_Exception) using the thread ID that I save from earlier and a generic exception type
PyGILState_Release()

So thank you to both the original question and to Peter Brittain!