Multiprocessing result of a psycopg2 request. “Can't pickle psycopg2.extensions.connection objects”

房东的猫 提交于 2020-04-17 19:25:46

问题


I'm currently trying to use multiprocessing to process a big result obtained after a psycopg2 query.

I split the result in several lists of 100 then send them for multiprocessing.

When I do so though, I get the following error

TypeError: can't pickle psycopg2.extensions.connection objects

Here is my psycopg2 query:

def get_employees(self):

    logging.info('POSTGRESQL QUERY: get_employees')

    try:
        self.cur = self.conn.cursor(cursor_factory=RealDictCursor)

        self.cur.execute(
            "SELECT ..."
        )

        employees = self.cur.fetchall()

    except (Exception, psycopg2.Error) as error:
        print("Error while connecting to PostgreSQL", error)
    finally:
        if self.conn:
            return employees

Here is how I do multiprocessing in this case:

self.db = database.ConnectionClass()
self.employees = self.db.get_employees()

pool = Pool()
for i in range(0, len(self.employees), 100):
    pool.apply_async(self.other_function, employees_sublists)

Thank you,

来源:https://stackoverflow.com/questions/58781116/multiprocessing-result-of-a-psycopg2-request-cant-pickle-psycopg2-extensions

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!