Python Multiprocessing with a single function

Question

I have a simulation that is currently running, but the ETA is about 40 hours -- I\'m trying to speed it up with multi-processing.

It essentially iterates over 3 valu

Answer 1

• Wrap the data for each iteration up into a tuple.
• Make a list data of those tuples
• Write a function f to process one tuple and return one result
• Create p = multiprocessing.Pool() object.
• Call results = p.map(f, data)

This will run as many instances of f as your machine has cores in separate processes.

Edit1: Example:

from multiprocessing import Pool

data = [('bla', 1, 3, 7), ('spam', 12, 4, 8), ('eggs', 17, 1, 3)]

def f(t):
    name, a, b, c = t
    return (name, a + b + c)

p = Pool()
results = p.map(f, data)
print results

Edit2:

Multiprocessing should work fine on UNIX-like platforms such as OSX. Only platforms that lack os.fork (mainly MS Windows) need special attention. But even there it still works. See the multiprocessing documentation.

Answer 2

Here is one way to run it in parallel threads:

import threading

L_a = []

for L in range(0,6,2):
    for a in range(1,100):
        L_a.append((L,a))
        # Add the rest of your objects here

def RunParallelThreads():
    # Create an index list
    indexes = range(0,len(L_a))
    # Create the output list
    output = [None for i in indexes]
    # Create all the parallel threads
    threads = [threading.Thread(target=simulate,args=(output,i)) for i in indexes]
    # Start all the parallel threads
    for thread in threads: thread.start()
    # Wait for all the parallel threads to complete
    for thread in threads: thread.join()
    # Return the output list
    return output

def simulate(list,index):
    (L,a) = L_a[index]
    list[index] = (a,L) # Add the rest of your objects here

master_list = RunParallelThreads()