Is there a good way to send data from python context to C++ without too much copy involved

问题

The title could be confusing. Here I will state my question more clearly.

I want to create a website based on python (a lot of existing framework like Flask and cherryPy) and along with a C++ computation engine for the sake of processing speed. So I need to create an interface for python to call C++ functions. Fortunately the boost.python can do the job. However, every time I send a data from python, say a matrix, to C++, I have to use python list, which means I have to transform the matrix data into list and in C++ context transform the list into an internal matrix object. As a result, a lot of data copy occur, which could not be an intelligent or efficient approach. So my questions is that if, considering the complexity, we don't map C++ matrix class to a python class through boost.python, is there a better way to do the similar work without or only with small number of copy?

回答1:

However, every time I send a data from python, say a matrix, to C++, I have to use python list, which means I have to transform the matrix data into list and in C++ context transform the list into an internal matrix object.

No, you don't have to use the python list. You can use numpy array which allocates data as a C contiguous segment which can be passed down to C++ without copying and viewed as a matrix using a matrix wrapper class.

In python allocate 2d array using numpy:

>>> y=np.empty((2,5), dtype=np.int16)
>>> y
array([[   12, 27750, 26465,  2675,     0],
       [    0,     0,     0,  2601,     0]], dtype=int16)
>>> y.flags['C_CONTIGUOUS']
True
>>> foo(y,2,5)

Pass matrix data to C++ using below function exposed to python:

void foo(python::object obj, size_t size1, size_t size2)
{
    PyObject* pobj = obj.ptr();
    Py_buffer pybuf;
    PyObject_GetBuffer(pobj, &pybuf, PyBUF_SIMPLE);
    void *buf = pybuf.buf;
    int16_t *p = (int16_t*)buf;
    Py_XDECREF(pobj);

    MyMatrixWrapper matrix(p, size1, size2);
    // ....
}

回答2:

There's a project called ndarray with this as their exact aim: https://github.com/ndarray/ndarray see also https://github.com/ndarray/Boost.NumPy .

There is some compatibility with other C++ matrix libraries (eg. Eigen) which should help with the computations.

来源：https://stackoverflow.com/questions/38883660/is-there-a-good-way-to-send-data-from-python-context-to-c-without-too-much-cop

标签

python

c++

boost-python

web-site-project