How to reinitialise an embedded Python interpreter?

Question

I'm working on embedding Python in our test suite application. The purpose is to use Python to run several tests scripts to collect data and make a report of tests. Multiple test scripts for one test run can create global variables and functions that can be used in the next script.

The application also provides extension modules that are imported in the embedded interpreter, and are used to exchange some data with the application.

But the user can also make multiple test runs. I don't want to share those globals, imports and the exchanged data between multiple test runs. I have to be sure I restart in a genuine state to control the test environment and get the same results.

How should I reinitialise the interpreter?

I used Py_Initialize() and Py_Finalize(), but get an exception on the second run when initialising a second time the extension modules I provide to the interpreter. And the documentation warns against using it more than once.

Using sub-interpreters seems to have the same caveats with extension modules initialization.

I suspect that I'm doing something wrong with the initialisation of my extension modules, but I fear that the same problem happens with 3rd party extension modules.

Maybe it's possible to get it to work by launching the interpreter in it's own process, so as to be sure that all the memory is released.

By the way, I'm using boost-python for it, that also warns AGAINST using Py_Finalize!

Any suggestion?

Thanks

Answer 1

Here is another way I found to achieve what I want, start with a clean slate in the interpreter.

I can control the global and local namespaces I use to execute the code:

// get the dictionary from the main module
// Get pointer to main module of python script
object main_module = import("__main__");
// Get dictionary of main module (contains all variables and stuff)
object main_namespace = main_module.attr("__dict__");

// define the dictionaries to use in the interpreter
dict global_namespace;
dict local_namespace;

// add the builtins
global_namespace["__builtins__"] = main_namespace["__builtins__"];

I can then use use the namespaces for execution of code contained in pyCode:

exec( pyCode, global_namespace, lobaca_namespace );

I can clean the namespaces when I want to run a new instance of my test, by cleaning the dictionaries:

// empty the interpreters namespaces
global_namespace.clear();
local_namespace.clear();        

// Copy builtins to new global namespace
global_namespace["__builtins__"] = main_namespace["__builtins__"];

Depending at what level I want the execution, I can use global = local

Answer 2

How about using code.IteractiveInterpreter?

Something like this should do it:

#include <boost/python.hpp>
#include <string>
#include <stdexcept>

using namespace boost::python;

std::string GetPythonError()
{
    PyObject *ptype = NULL, *pvalue = NULL, *ptraceback = NULL;
    PyErr_Fetch(&ptype, &pvalue, &ptraceback);
    std::string message("");
    if(pvalue && PyString_Check(pvalue)) {
        message = PyString_AsString(pvalue);
    }
    return message;
}

// Must be called after Py_Initialize()
void RunInterpreter(std::string codeToRun)
{
    object pymodule = object(handle<>(borrowed(PyImport_AddModule("__main__"))));
    object pynamespace = pymodule.attr("__dict__");

    try {
        // Initialize the embedded interpreter
        object result = exec(   "import code\n"
                                "__myInterpreter = code.InteractiveConsole() \n", 
                                pynamespace);
        // Run the code
        str pyCode(codeToRun.c_str());
        pynamespace["__myCommand"] = pyCode;
        result = eval("__myInterpreter.push(__myCommand)", pynamespace);
    } catch(error_already_set) {
        throw std::runtime_error(GetPythonError().c_str());
    }
}

Answer 3

I'd write another shell script executing the sequence of test scripts with new instances of python each time. Or write it in python like

# run your tests in the process first
# now run the user scripts, each in new process to have virgin env
for script in userScript:
    subprocess.call(['python',script])