multiprocessing

问题 I am trying to send several parallel requests to a XMLRPC server (mosesserver). To launch the XMLRPC call, a ServerProxy object is needed (this object contains the URL of the server, among other things) In serial execution, I can create this object at the beginning of the program (with a call to xmlrpclib.ServerProxy(server_URL)), store it in a variable and use it whenever I need it. But when using a pool of processes, each process needs a different instance of this object. The task of each

问题 I have to do a program that read and image and puts it into a byte array var Imagenoriginal = File.ReadAllBytes("10M.bmp"); And Divide That byte Array into 3 Diferent Arrays in order to send each one of this new arrays to other computer ( Using Pipes ) to process them there and finally take them back to the origial computer and finally give the result. But my question Is how do I do an Algorithm able to divide the byte array in three different bytes arrays if the image selected can have

问题 $ python2.6 -c 'from multiprocessing.queues import SimpleQueue' $ python2.7 -c 'from multiprocessing.queues import SimpleQueue' Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python2.7/multiprocessing/queues.py", line 22, in <module> from multiprocessing.synchronize import Lock, BoundedSemaphore, Semaphore, Condition File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 33, in <module> " function, see issue 3770.") ImportError: This platform

问题 I couldn't find a better way to describe the error I'm facing, but this error seems to come up everytime I try to implement Multiprocessing to a loop call. I've used both sklearn.externals.joblib as well as multiprocessing.Process but error are similar though different. Original Loop on which want to apply Multiprocessing, where one iteration in executed in single thread/process for dd in final_col_dates: idx1 = final_col_dates.tolist().index(dd) dataObj = GetPrevDataByDate(d1, a, dd, self

问题 Please observe this simple code: import random while True: L = list( str(random.random())) Question: if I let this run, will python run out of memory? reason I am asking: First iteration of this loop, a list is created, and 'L' is assigned to represent that list. The next iteration of this loop, another list is created, 'L' is yanked from the previous list and and assigned to the new list. The previous list has lost it reference. Is the previous list going to be garbage collected? if not at

问题 I have to make numerous (thousands) of HTTP GET requests to a great deal of websites. This is pretty slow, for reasons that some websites may not respond (or take long to do so), while others time out. As I need as many responses as I can get, setting a small timeout (3-5 seconds) is not in my favour. I have yet to do any kind of multiprocessing or multi-threading in Python, and I've been reading the documentation for a good while. Here's what I have so far: import requests from bs4 import

问题 I'm getting this error when doing database calls in a sub process using multiprocessing library. Visit : Pastie InternalError: current transaction is aborted, commands ignored until end of transaction block this is to a Postgre Database, using psycopg2 driver in web.py . However if I use threading.Thread instead of multiprocessing.Process I don't get this error. Any idea how to fix this? 回答1: multiprocessing works (on UNIX systems) by forking the current process. If you have an existing

问题 I'm testing out how multiprocessing works and would like an explanation why I'm getting this exception and if it is even possible to pass the sqlite3 Connection Object this way: import sqlite3 from multiprocessing import Queue, Process def sql_query_worker(conn, query_queue): # Creating the Connection Object here works... #conn = sqlite3.connect('test.db') while True: query = query_queue.get() if query == 'DO_WORK_QUIT': break c = conn.cursor() print('executing query: ', query) c.execute

问题 I am using python multiprocessing to split one of the longer processes and run parallelly. It is working fine except when there is an exception in one of the child processes, in which case, process pool is not closed and I can still see those processes on the server. Here is the code: from multiprocessing import Pool pool = Pool(processes=4) from functools import partial param_data = "Test Value" func = partial(test_function, param_data) r = pool.map(func, range(3)) pool.close() def test

问题 I have code that, simplified down, looks like this: run = functools.partial(run, grep=options.grep, print_only=options.print_only, force=options.force) if not options.single and not options.print_only and options.n > 0: pool = multiprocessing.Pool(options.n) Map = pool.map else: Map = map for f in args: with open(f) as fh: Map(run, fh) try: pool.close() pool.join() except NameError: pass That works fine when I run it in single process mode, but fails with errors like this TypeError: type