multiprocessing

Python multiprocessing with start method 'spawn' doesn't work

荒凉一梦 提交于 2020-08-26 10:12:26
问题 I wrote a Python class to plot pylots in parallel. It works fine on Linux where the default start method is fork but when I tried it on Windows I ran into problems (which can be reproduced on Linux using the spawn start method - see code below). I always end up getting this error: Traceback (most recent call last): File "test.py", line 50, in <module> test() File "test.py", line 7, in test asyncPlotter.saveLinePlotVec3("test") File "test.py", line 41, in saveLinePlotVec3 args=(test, )) File

A ThreadPoolExecutor inside a ProcessPoolExecutor

倾然丶 夕夏残阳落幕 提交于 2020-08-21 09:44:08
问题 I am new to the futures module and have a task that could benefit from parallelization; but I don't seem to be able to figure out exactly how to setup the function for a thread and the function for a process. I'd appreciate any help anyone can shed on the matter. I'm running a particle swarm optimization (PSO). Without getting into too much detail about PSO itself, here's the basic layout of my code: There is a Particle class, with a getFitness(self) method (which computes some metric and

Tracking progress of joblib.Parallel execution

末鹿安然 提交于 2020-08-21 05:02:06
问题 Is there a simple way to track the overall progress of a joblib.Parallel execution? I have a long-running execution composed of thousands of jobs, which I want to track and record in a database. However, to do that, whenever Parallel finishes a task, I need it to execute a callback, reporting how many remaining jobs are left. I've accomplished a similar task before with Python's stdlib multiprocessing.Pool, by launching a thread that records the number of pending jobs in Pool's job list.