Python embedded in CPP: how to get data back to CPP

Question

While working on a C++ project, I was looking for a third party library for something that is not my core business. I found a really good library, doing exactly what's needed, but it is written in Python. I decided to experiment with embedding Python code in C++, using the Boost.Python library.

The C++ code looks something like this:

#include <string>
#include <iostream>
#include <boost/python.hpp>

using namespace boost::python;

int main(int, char **) 
{
    Py_Initialize();

    try 
    {
        object module((handle<>(borrowed(PyImport_AddModule("__main__")))));

        object name_space = module.attr("__dict__");
        object ignored = exec("from myModule import MyFunc\n"
                          "MyFunc(\"some_arg\")\n",
                          name_space);

        std::string res = extract<std::string>(name_space["result"]);
    } 
    catch (error_already_set) 
    {
        PyErr_Print();
    }

    Py_Finalize();
    return 0;
}

A (very) simplified version of the Python code looks like this:

import thirdparty

def MyFunc(some_arg):
    result = thirdparty.go()
    print result

Now the problem is this: 'MyFunc' executes fine, i can see the print of 'result'. What i cannot do is read 'result' back from the C++ code. The extract command never finds 'result' in any namespace. I tried defining 'result' as a global, i even tried returning a tuple, but i cannot get it to work.

Answer 1

First of all, change your function to return the value. printing it will complicate things since you want to get the value back. Suppose your MyModule.py looks like this:

import thirdparty

def MyFunc(some_arg):
    result = thirdparty.go()
    return result

Now, to do what you want, you have to go beyond basic embedding, as the documentation says. Here is the full code to run your function:

#include <Python.h>

int
main(int argc, char *argv[])
{
    PyObject *pName, *pModule, *pFunc;
    PyObject *pArgs, *pArg, *pResult;
    int i;

    Py_Initialize();
    pName = PyString_FromString("MyModule.py");
    /* Error checking of pName left out as exercise */

    pModule = PyImport_Import(pName);
    Py_DECREF(pName);

    if (pModule != NULL) {
        pFunc = PyObject_GetAttrString(pModule, "MyFunc");
        /* pFunc is a new reference */

        if (pFunc) {
            pArgs = PyTuple_New(0);
            pArg = PyString_FromString("some parameter")
            /* pArg reference stolen here: */
            PyTuple_SetItem(pArgs, 0, pArg);
            pResult = PyObject_CallObject(pFunc, pArgs);
            Py_DECREF(pArgs);
            if (pResult != NULL) {
                printf("Result of call: %s\n", PyString_AsString(pResult));
                Py_DECREF(pResult);
            }
            else {
                Py_DECREF(pFunc);
                Py_DECREF(pModule);
                PyErr_Print();
                fprintf(stderr,"Call failed\n");
                return 1;
            }
        }
        else {
            if (PyErr_Occurred())
                PyErr_Print();
            fprintf(stderr, "Cannot find function");
        }
        Py_XDECREF(pFunc);
        Py_DECREF(pModule);
    }
    else {
        PyErr_Print();
        fprintf(stderr, "Failed to load module");
        return 1;
    }
    Py_Finalize();
    return 0;
}

Answer 2

Based on ΤΖΩΤΖΙΟΥ, Josh and Nosklo's answers i finally got it work using boost.python:

Python:

import thirdparty

def MyFunc(some_arg):
    result = thirdparty.go()
    return result

C++:

#include <string>
#include <iostream>
#include <boost/python.hpp>

using namespace boost::python;

int main(int, char **) 
{
    Py_Initialize();

    try 
    {
        object module = import("__main__");
        object name_space = module.attr("__dict__");
        exec_file("MyModule.py", name_space, name_space);

        object MyFunc = name_space["MyFunc"];
        object result = MyFunc("some_args");

        // result is a dictionary
        std::string val = extract<std::string>(result["val"]);
    } 
    catch (error_already_set) 
    {
        PyErr_Print();
    }

    Py_Finalize();
    return 0;
}

Some important points:

1. I changed 'exec' to 'exec_file' out of convenience, it also works with plain 'exec'.
2. The main reason it failed is that i did not pass a "local" name_sapce to 'exec' or 'exec_file' - this is now fixed by passing name_space twice.
3. If the python function returns unicode strings, they are not convertible to 'std::string', so i had to suffix all python strings with '.encode('ASCII', 'ignore')'.

Answer 3

I think what you need is either PyObject_CallObject(<py function>, <args>), which returns the return value of the function you call as a PyObject, or PyRun_String(<expression>, Py_eval_input, <globals>, <locals>) which evaluates a single expression and returns its result.

Answer 4

You should be able to return the result from MyFunc, which would then end up in the variable you are currently calling "ignored". This eliminates the need to access it in any other way.