How to combine multiprocessing and eventlet

问题

I have a task need to start 2 processes and within each process need to start 2 threads to do really work. Below is the source code I used to simulate my use case.

import multiprocessing
import eventlet

def subworker(num1, num2):
    print 'Start subworker %d,%d\n' % (num1, num2)
    eventlet.sleep(10)
    print 'End subworker %d,%d\n' % (num1, num2)

def worker(**kwargs):
    number = kwargs['number']
    pool = eventlet.GreenPool(size=2)
    pool.spawn_n(subworker, number, 1)
    pool.spawn_n(subworker, number, 2)
    pool.waitall()

def launcher(number):
    kwargs = {'number': number}
    th = multiprocessing.Process(target=worker, kwargs=kwargs)
    th.start()
    while True:
        if not th.is_alive():
            break
        eventlet.sleep(0)

    th.join()


def main():
    pool = eventlet.GreenPool(size=2)
    pool.spawn_n(launcher, 1)
    pool.spawn_n(launcher, 2)
    pool.waitall()

main()

When I run this python script my expected output is something like:

Start subworker 1,1 
Start subworker 1,2
Start subworker 2,1
Start subworker 2,2
End subworker 1,1
End subworker 1,2
End subworker 2,1
End subworker 2,2

But what I really got is:

Start subworker 1,1
Traceback (most recent call last):

  File "/Users/leehom/python_local/lib/python2.7/site-packages/eventlet/greenpool.py", line 82, in _spawn_n_impl
Start subworker 1,2
    func(*args, **kwargs)

  File "/Users/leehom/Desktop/home/work_dir/source/snips/Test_multiprocessing_and_eventlet.py", line 27, in launcher
Start subworker 2,1
    if not th.is_alive():

  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 155, in is_alive
Start subworker 2,2
    assert self._parent_pid == os.getpid(), 'can only test a child process'

AssertionError: can only test a child process
Start subworker 1,1
Start subworker 1,2
Start subworker 2,1
Start subworker 2,2
End subworker 1,1
End subworker 1,2
End subworker 2,1
End subworker 2,2
End subworker 1,1
End subworker 1,2
End subworker 2,1
End subworker 2,2
Process finished with exit code 0

Seems launcher was called twice. I do not understand why this happen.

If I comment one line in my main function

def main():
    pool = eventlet.GreenPool(size=2)
    pool.spawn_n(launcher, 1)
    # pool.spawn_n(launcher, 2)
    pool.waitall()

The result is right:

Start subworker 1,1
Start subworker 1,2
End subworker 1,1
End subworker 1,2

Anyone knows how I can fix this and why this problem happen?

回答1:

As of 2018-01, Eventlet and multiprocessing don't work well together. Your best option is to spawn worker processes externally. Second best option is os.fork() to create worker processes and only then import eventlet.

Subscribe to this issue to be notified when multiprocessing compatibility is resolved. https://github.com/eventlet/eventlet/issues/147

回答2:

The main module should be importable for multiple process to work properly. Do not call main() in global space, use it like:

if name == 'main': main()

来源：https://stackoverflow.com/questions/48218396/how-to-combine-multiprocessing-and-eventlet

标签

python

multiprocessing

eventlet