Python multiprocessing - Independently processing for each key-value pair in the dictionary

问题

I have a dictionary that looks like this:

sampleData = {'x1': [1,2,3], 'x2': [4,5,6], 'x3': [7,8,9]}

I need to do some calculation for each key and value pair by passing data to a blackBoxFunction. This function takes time to do the processing. The final output is stored in a separate dictionary finalValue = {}.

This is the code for doing it sequentially:

for key in sampleData.keys():
    finalValue[key] = []
    for i in range(0,len(sampleData[key])):
        for j in range(i,len(sampleData[key])):
            if(i!=j):
                finalValue[key].append(blackBoxFunction(sampleData[key][i],sampleData[key][j]))

However, the calculation for each key and value pair is independent of each other.

I want to know how can I use multiprocessing library in my code to perform the task in parallel.

The final dictionary structure will look similar to the input dictionary.

finalValue 
{'x1': [31, 43, 53], 'x2': [97, 110, 131], 'x3': [135, 164, 137]}

回答1:

Try something along the lines

from multiprocessing import Pool

def pair_black_box(data):
    key, values = data
    res = []
    for i in range(0, len(values)):
        for j in range(i, len(values)):
            if(i != j):
               res.append(blackBoxFunction(values[i], values[j]))
    return key, res

p = Pool(3)
finalValue = dict(p.map(pair_black_box, sampleData.items()))

来源：https://stackoverflow.com/questions/53937320/python-multiprocessing-independently-processing-for-each-key-value-pair-in-the