multiprocessing

问题 I observed this behavior when trying to create nested child processes in Python. Here is the parent program parent_process.py : import multiprocessing import child_process pool = multiprocessing.Pool(processes=4) for i in range(4): pool.apply_async(child_process.run, ()) pool.close() pool.join() The parent program calls the "run" function in the following child program child_process.py: import multiprocessing def run(): pool = multiprocessing.Pool(processes=4) print 'TEST!' pool.close() pool

问题 If a software project supports a version of Python that multiprocessing has been backported to, is there any reason to use threading.Lock over multiprocessing.Lock ? Would a multiprocessing lock not be thread safe as well? For that matter, is there a reason to use any synchronization primitives from threading that are also in multiprocessing ? 回答1: The threading module's synchronization primitive are lighter and faster than multiprocessing, due to the lack of dealing with shared semaphores,

问题 I'm attempting to build a python script that has a pool of worker processes (using mutiprocessing.Pool) across a large set of data. I want each process to have a unique object that gets used across multiple executes of that process. Psudo code: def work(data): #connection should be unique per process connection.put(data) print 'work done with connection:', connection if __name__ == '__main__': pPool = Pool() # pool of 4 processes datas = [1..1000] for process in pPool: #this is the part i'm

问题 I am confused with Python multiprocessing. I am trying to speed up a function which process strings from a database but I must have misunderstood how multiprocessing works because the function takes longer when given to a pool of workers than with “normal processing”. Here an example of what I am trying to achieve. from time import clock, time from multiprocessing import Pool, freeze_support from random import choice def foo(x): TupWerteMany = [] for i in range(0,len(x)): TupWerte = [] s =

问题 I have an idea. Write a WebSocket based RPC that would process messages according to the scenario below. Client connects to a WS (web socket) server Client sends a message to the WS server WS server puts the message into the incoming queue (can be a multiprocessing.Queue or RabbitMQ queue) One of the workers in the process pool picks up the message for processing Message is being processed (can be blazingly fast or extremely slow - it is irrelevant for the WS server) After the message is

问题 I have a simple python multiprocessing script that sets up a pool of workers that attempt to append work-output to a Manager list. The script has 3 call stacks: - main calls f1 that spawns several worker processes that call another function g1. When one attempts to debug the script (incidentally on Windows 7/64 bit/VS 2010/PyTools) the script runs into a nested process creation loop, spawning an endless number of processes. Can anyone determine why? I'm sure I am missing something very simple

问题 I just tried a Python shelve module as the persistent cache for data fetched from the external service. The complete example is here. I was wondering what would the best approach if I want to make this multiprocess safe? I am aware of redis, memcached and such "real solutions", but I'd like to use only the parts of Python standard library or very minimal dependencies to keep my code compact and not introduce unnecessary complexity when running the code in single process - single thread model.

问题 I have a method inside a class that needs to do a lot of work in a loop, and I would like to spread the work over all of my cores. I wrote the following code, which works if I use normal map , but with pool.map returns an error. import multiprocessing pool = multiprocessing.Pool(multiprocessing.cpu_count() - 1) class OtherClass: def run(sentence, graph): return False class SomeClass: def __init__(self): self.sentences = [["Some string"]] self.graphs = ["string"] def some_method(self): other =

问题 I have a method inside a class that needs to do a lot of work in a loop, and I would like to spread the work over all of my cores. I wrote the following code, which works if I use normal map , but with pool.map returns an error. import multiprocessing pool = multiprocessing.Pool(multiprocessing.cpu_count() - 1) class OtherClass: def run(sentence, graph): return False class SomeClass: def __init__(self): self.sentences = [["Some string"]] self.graphs = ["string"] def some_method(self): other =

问题 tl;dr How can I serialise a Flask app or request context, or a subset of that context (i.e. whatever can be successfully serialised) so that I can access that context from another process, rather than a thread? Long version I have some functions that require access to the Flask request context, or the App context, that I want to run in the background. Flask has a built-in @copy_current_request_context decorator to wrap a function in a copy of the request context, so you can run it in a