multiprocessing

问题 I have code that, simplified down, looks like this: run = functools.partial(run, grep=options.grep, print_only=options.print_only, force=options.force) if not options.single and not options.print_only and options.n > 0: pool = multiprocessing.Pool(options.n) Map = pool.map else: Map = map for f in args: with open(f) as fh: Map(run, fh) try: pool.close() pool.join() except NameError: pass That works fine when I run it in single process mode, but fails with errors like this TypeError: type

问题 I'm developing a realtime software that I have some non real time functions. Specs: Need do the method1 in real time; Need do the method2 each 60 minutes. Should I use multithreading? Tasks? Now I'm using timers but don't think that it's a good use for it. 回答1: It is generally a bad idea to have file processing start immediately a file appears in a folder. For example if 100 files arrive at once then you will end up with 100 threads all competing for the same physical resources, e.g. disk

问题 I am trying Threading and Multiprocessing for python on a windows machine.But python give the following message. RuntimeError: Attempt to start a new process before the current process has finished its bootstrapping phase. This probably means that you are on Windows and you have forgotten to use the proper idiom in the main module: if __name__ == '__main__': freeze_support() ... The "freeze_support()" line can be omitted if the program is not going to be frozen to produce a Windows executable

问题 I have an application that reads a series of XML files containing logs of vehicles passages in a road. The application then processes each record, transform a few of the informations to match the database columns and inserts it into a cassandra database (running a single node in a remote server [it's in an internal network so connection isn't really an issue]). After inserting data in the database, the process for each file then goes on to read this data and produce information for summary

问题 For an internship on the Python library fluidimage, we are investigating if it could be a good idea to write a HPC parallel application with a client/servers model using the library trio. For asynchronous programming and i/o, trio is indeed great! Then, I'm wondering how to spawn processes (the servers doing the CPU-GPU bounded work) communicating complex Python objects (potentially containing large numpy arrays) between the processes. I didn't find what was the recommended way to do this

问题 I am using Python's multiprocessing to create a parallel application. Processes need to share some data, for which I use a Manager . However, I have some common functions which processes need to call and which need to access the data stored by the Manager object. My question is whether I can avoid needing to pass the Manager instance to these common functions as an argument and rather use it like a global. In other words, consider the following code: import multiprocessing as mp manager = mp

问题 I am using Python's multiprocessing to create a parallel application. Processes need to share some data, for which I use a Manager . However, I have some common functions which processes need to call and which need to access the data stored by the Manager object. My question is whether I can avoid needing to pass the Manager instance to these common functions as an argument and rather use it like a global. In other words, consider the following code: import multiprocessing as mp manager = mp

问题 I’d like to skip results that are returned from map_async. They are growing in memory but I don’t need them. Here is some code: def processLine(line): #process something print "result" pool = Pool(processes = 8) for line in sys.stdin: lines.append(line) if len(lines) >= 100000: pool.map_async(processLine, lines, 2000) pool.close() pool.join() When I have to process file with hundreds of millions of rows, the python process grows in memory to a few gigabytes. How can I resolve that? Thanks for

问题 I’d like to skip results that are returned from map_async. They are growing in memory but I don’t need them. Here is some code: def processLine(line): #process something print "result" pool = Pool(processes = 8) for line in sys.stdin: lines.append(line) if len(lines) >= 100000: pool.map_async(processLine, lines, 2000) pool.close() pool.join() When I have to process file with hundreds of millions of rows, the python process grows in memory to a few gigabytes. How can I resolve that? Thanks for

问题 I have trouble understanding why this simple program raises an EOFError at the end. I am using a Queue() to communicate with a Thread() that I want to automatically and cleanly terminate atexit of my program. import threading import multiprocessing import atexit class MyClass: def __init__(self): self.queue = None self.thread = None def start(self): self.queue = multiprocessing.Queue() self.thread = threading.Thread(target=self.queued_writer, daemon=True) self.thread.start() # Remove this: no