Pybind11: Accessing python object with OpenMP using for-loop

问题

I am trying to operate a c++ function on all elements of a python dictionary. For this I use a for loop in c++ over all elements of the dictionary. In this case and as far as I understand, this can be sped up using the #pragma omp parallel for simd clause. However, when I run it, I get the error:

GC Object already Tracked
Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)

Edit

I have read on this post that the problem comes from the way of accessing a Python object in c++, however it does not go into further discussion. I guess the problem comes from accessing a python object with multiple threads. I am still currently searching.

Any hint would be highly appreciated.

Thanks a lot in advance.

C++ Code

#include <pybind11/pybind11.h>
#include <pybind11/numpy.h>
#include <pybind11/stl.h>
#include <omp.h>
#include <chrono>
#include <thread>

namespace py = pybind11;
py::module nn = py::module::import("neat.nn");

py::object create_seq(
  py::object self
  ){

  unsigned int z;
  #pragma omp parallel for simd
  for(z = 0; z < 50000; z++){
      double value = self.attr("dict").attr("__getitem__")(z).cast<double>();
      value *=2;
      self.attr("dict").attr("__setitem__")(z, value);
  }
  return self;
}

PYBIND11_MODULE(error, m){

    m.doc() = "pybind11 module for iterating over generations";

    m.def("create_seq", &create_seq,
      "the function which creates a sequence");

}

Python code

import os
import error

class func():
    def __init__(self):

        dictionary = {}
        for i in range(50000):
            dictionary[i] = i

        self.dict = dictionary
        self.array = None

    def modify_dict(self):
        return error.create_seq(self)

if __name__ == '__main__':
    # Determine path to configuration file. This path manipulation is
    # here so that the script will run successfully regardless of the
    # current working directory.
    local_dir = os.path.dirname(__file__)
    a = func()
    a.modify_dict()
    print(a.dict)

Compiled with:

g++ -O3 -Wall -shared -std=c++14 -fopenmp -fPIC `python3 -m pybind11 --includes` openmp.cpp -o error.so

回答1:

The problem is indeed accessing the python object with multiple threads. I don't know the exact reasons why, but this is simply impossible. The trick is to transform your python object into a list of python objects, which must be converted into a vector. Only then, the multiple threads access the various python objects of the vector.

In my case, I passed not the entire dictionary rather than the values of the dictionary in a list, and then converted this list into a vector of python objects.

In resumé it is something like:

error.create_seq(list(dict.values()))

and in c++:

std::vector<py::object> = dict.cast<std::vector<py::object>>();

and then the #pragma omp parallel clause can be applied

来源：https://stackoverflow.com/questions/59695831/pybind11-accessing-python-object-with-openmp-using-for-loop

标签

python

c++

openmp

pybind11