multiprocessing

问题 I'm using DEAP library to implement genetic programming and I have used eaMuCommaLambda algorithm for this purpose. In order to run the program in parallel, I followed the instructions in the DEAP document and added the two following lines of code in the if __name__ == "__main__" section. import multiprocessing pool = multiprocessing.Pool() toolbox.register("map", pool.map) pop, log = algorithms.eaMuCommaLambda(pop, toolbox, MU, LAMBDA, cxpb, mutpb, gen, halloffame=hof, stats=mstats, verbose

问题 I'm using DEAP library to implement genetic programming and I have used eaMuCommaLambda algorithm for this purpose. In order to run the program in parallel, I followed the instructions in the DEAP document and added the two following lines of code in the if __name__ == "__main__" section. import multiprocessing pool = multiprocessing.Pool() toolbox.register("map", pool.map) pop, log = algorithms.eaMuCommaLambda(pop, toolbox, MU, LAMBDA, cxpb, mutpb, gen, halloffame=hof, stats=mstats, verbose

问题 I need to run many processes, but not all together, for example 4 processes at same time. multiprocessing.Pool is exactly what I need. But the problem is that I need to terminate a process if it lasts more than a timeout (e.g. 3 seconds). Pool just supports wait for a timeout for all processes not each of them. This is what I need: def f(): process_but_kill_if_it_takes_more_than_3_sec() pool.map(f, inputs) I couldn't find a simple way to use Pool with timeouts. There is a solution from Eli

问题 I need to run many processes, but not all together, for example 4 processes at same time. multiprocessing.Pool is exactly what I need. But the problem is that I need to terminate a process if it lasts more than a timeout (e.g. 3 seconds). Pool just supports wait for a timeout for all processes not each of them. This is what I need: def f(): process_but_kill_if_it_takes_more_than_3_sec() pool.map(f, inputs) I couldn't find a simple way to use Pool with timeouts. There is a solution from Eli

问题 I want to share a numpy array across multiple processes. The processes only read the data, so I want to avoid making copies. I know how to do it if I can start with a multiprocessing.sharedctypes.RawArray and then create a numpy array using numpy.frombuffer . But what if I am initially given a numpy array? Is there a way to initialize a RawArray with the numpy array's data without copying the data? Or is there another way to share the data across the processes without copying it? 回答1: I also

问题 First look at the following code: pool = multiprocessing.Pool(processes=N) batch = [] for item in generator(): batch.append(item) if len(batch) == 10: pool.apply_async(my_fun, args=(batch,)) batch = [] # leftovers pool.apply_async(my_fun, args=(batch,)) Essentially I'm retrieving data from a generator, collecting in into a list and then spawning a process that consumes the batch of data. This may look fine but when the consumers (aka the pool processes) are slower than the producer (aka the

问题 Is it possible to have a few child processes running some calculations, then send the result to main process (e.g. update PyQt ui), but the processes are still running, after a while they send back data and update ui again? With multiprocessing.queue, it seems like the data can only be sent back after process is terminated. So I wonder whether this case is possible or not. 回答1: I don't know what you mean by "With multiprocessing.queue, it seems like the data can only be sent back after

问题 Is it possible to have a few child processes running some calculations, then send the result to main process (e.g. update PyQt ui), but the processes are still running, after a while they send back data and update ui again? With multiprocessing.queue, it seems like the data can only be sent back after process is terminated. So I wonder whether this case is possible or not. 回答1: I don't know what you mean by "With multiprocessing.queue, it seems like the data can only be sent back after

问题 Is it possible to have a few child processes running some calculations, then send the result to main process (e.g. update PyQt ui), but the processes are still running, after a while they send back data and update ui again? With multiprocessing.queue, it seems like the data can only be sent back after process is terminated. So I wonder whether this case is possible or not. 回答1: I don't know what you mean by "With multiprocessing.queue, it seems like the data can only be sent back after

问题 Is it possible to have a few child processes running some calculations, then send the result to main process (e.g. update PyQt ui), but the processes are still running, after a while they send back data and update ui again? With multiprocessing.queue, it seems like the data can only be sent back after process is terminated. So I wonder whether this case is possible or not. 回答1: I don't know what you mean by "With multiprocessing.queue, it seems like the data can only be sent back after