Share same multiprocessing.Pool object between different python instances

问题

In Python 3 I need to have a Pool of processes in which, asynchronously, apply multiple workers.

The problem is that I need to "send" workers to the Pool from a series of separate Python processes. So, all the worker should be executed in the same Pool instance.

N.B. The objective is to process a lot of data without use all the computer resources.

Having the following multi.py example code:

import multiprocessing
from time import sleep

def worker(x):
    sleep(5)
    return x*x

if __name__ == "__main__":
    pool = multiprocessing.Pool(processes=int(multiprocessing.cpu_count()/2)) # Using half of the CPU cores

    for i in range(10):
        pool.apply_async(worker, args=(i,))

I need, opening multiple multi.py instances, to append workers to the same pool.

Reading the official documentation I cannot understand a way to do this. I understood I would need a Manager() but how should use it?

Any advice for this in a Python-way or anyone having a working piece of code?

Thanks you all.

回答1:

In the end I was able to code a working basic example using Python 3 BaseManager. See docs here.

In a script called server.py:

jobs = multiprocessing.Manager().Queue()
BaseManager.register('JobsQueue', callable = lambda: jobs)
m = BaseManager(address=('localhost', 55555), authkey=b'myauthkey')
s = m.get_server()
s.serve_forever()

Then in one or more scripts client.py:

BaseManager.register('JobsQueue') # See the difference with the server!
m = BaseManager(address=('localhost', 55555), authkey=b'myauthkey') # Use same authkey! It may work remotely too...
m.connect()
# Then you can put data in the queue
q = m.JobsQueue()
q.put("MY DATA HERE")
# or also
data = q.get()
# etc etc...

Obviously this is a basic example, but I think it offers to do a lot of complex work without using external libraries.

A lot of people today look to a ready to use, often massive weight, library or software, without understand the basics. I'm not one of them...

Cheers

来源：https://stackoverflow.com/questions/49521479/share-same-multiprocessing-pool-object-between-different-python-instances

标签

python

python-3.x

multiprocessing