问题
I'm currently trying to use multiprocessing to process a big result obtained after a psycopg2 query.
I split the result in several lists of 100 then send them for multiprocessing.
When I do so though, I get the following error
TypeError: can't pickle psycopg2.extensions.connection objects
Here is my psycopg2 query:
def get_employees(self):
logging.info('POSTGRESQL QUERY: get_employees')
try:
self.cur = self.conn.cursor(cursor_factory=RealDictCursor)
self.cur.execute(
"SELECT ..."
)
employees = self.cur.fetchall()
except (Exception, psycopg2.Error) as error:
print("Error while connecting to PostgreSQL", error)
finally:
if self.conn:
return employees
Here is how I do multiprocessing in this case:
self.db = database.ConnectionClass()
self.employees = self.db.get_employees()
pool = Pool()
for i in range(0, len(self.employees), 100):
pool.apply_async(self.other_function, employees_sublists)
Thank you,
来源:https://stackoverflow.com/questions/58781116/multiprocessing-result-of-a-psycopg2-request-cant-pickle-psycopg2-extensions