AssertionError using pathos.multiprocessing

生来就可爱ヽ(ⅴ<●) 提交于 2019-12-11 06:26:24

问题


I'm using python 2.7. Here's some of my code:

from pathos.multiprocessing import ProcessingPool as Pool

class pair_scraper(object):

    #code and various methods ...

    def parallelized_processing(self):
    '''
    Parallelize length extraction of
    relevant files.
    '''
    pool = Pool(self.pool_size)
    pool.map(self.length_processer, self.zip_files)
    pool.close()
    pool.join()

However when I run this, I get a blank AssertionError:

/...file.py in parallelized_processing(self)
     69         '''
     70         pool = Pool(self.pool_size)
---> 71         pool.map(self.length_processer, self.zip_files)
     72         # pool.close()
     73         # pool.join()

/.../anaconda/lib/python2.7/site-packages/pathos/multiprocessing.pyc in map(self, f, *args, **kwds)
    134         AbstractWorkerPool._AbstractWorkerPool__map(self, f, *args, **kwds)
    135         _pool = self._serve()
--> 136         return _pool.map(star(f), zip(*args)) # chunksize
    137     map.__doc__ = AbstractWorkerPool.map.__doc__
    138     def imap(self, f, *args, **kwds):

/.../anaconda/lib/python2.7/site-packages/multiprocess/pool.pyc in map(self, func, iterable, chunksize)
    248         Equivalent of `map()` builtin
    249         '''
--> 250         assert self._state == RUN
    251         return self.map_async(func, iterable, chunksize).get()
    252 

AssertionError: 

This error seems to resemble the one described in this question, although the fix there was more obvious. I ran the above code removing pool.close() and pool.join() but got the same results.

Thoughts?

来源:https://stackoverflow.com/questions/40734063/assertionerror-using-pathos-multiprocessing

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!