Parallel execution of class methods

Question

I need to execute in parallel a method of many instances of the same class. For doing this I\'m trying to use the Process.start() and the Process.join() commands from the mu

Answer 1

When you call obj.method in a child process, the child process is getting its own separate copy of each instance variable in obj. So, the changes you make to them in the child will not be reflected in the parent. You'll need to explicitly pass the changed values back to the parent via a multiprocessing.Queue in order to make the changes take effect the parent:

from multiprocessing import Process, Queue
q1 = Queue()
q2 = Queue()
pr1 = Process(target=t1.method, args=(q1,))
pr2 = Process(target=t2.method, args=(q2,))
pr1.start()
pr2.start()
out1 = q1.get()
out2 = q2.get()
t1.blah = out1
t2.blah = out2
pr1.join()
pr2.join()

Other options would be to make the instance variables you need to change multiprocessing.Value instances, or multiprocessing.Manager Proxy instances. That way, the changes you make in the children would be reflected in the parent automatically. But that comes at the cost of adding overhead to using the variables in the parent.

Here's an example using multiprocessing.Manager. This doesn't work:

import multiprocessing

class Test(object) :

    def __init__(self):
       self.some_list = []  # Normal list

    def method(self):
        self.some_list.append(123)  # This change gets lost


if __name__ == "__main__":
    t1 = Test()
    t2 = Test()
    pr1 = multiprocessing.Process(target=t1.method)
    pr2 = multiprocessing.Process(target=t2.method)
    pr1.start()
    pr2.start()
    pr1.join()
    pr2.join()
    print(t1.some_list)
    print(t2.some_list)

Output:

[]
[]

This works:

import multiprocessing

class Test(object) :

    def __init__(self):
       self.manager = multiprocessing.Manager()
       self.some_list = self.manager.list()  # Shared Proxy to a list

    def method(self):
        self.some_list.append(123) # This change won't be lost


if __name__ == "__main__":
    t1 = Test()
    t2 = Test()
    pr1 = multiprocessing.Process(target=t1.method)
    pr2 = multiprocessing.Process(target=t2.method)
    pr1.start()
    pr2.start()
    pr1.join()
    pr2.join()
    print(t1.some_list)
    print(t2.some_list)

Output:

[123]
[123]

Just keep in mind that a multiprocessing.Manager starts a child process to manage all the shared instances you create, and that every time you access one of the Proxy instances, you're actually making an IPC call to the Manager process.