How to unpack results from `Pool.map()`?

问题

I've got a function (preprocess_fit) that will first preprocess a data set (i.e. smoothing, baseline correction, and filters out bad data). The function then takes an initial guess for a parameter and then iterates through guesses to find the optimised fit and then returns const1, const2. The function also calculates a bunch of other parameters but they are not returned in this instance.

I then need to loop this function over all files in a directory (~1000 files). I do this by using the second function (function) that contains a for loop. The preprocessing step, in particular the guess iteration is particularly time consuming.

I'd like to pool the function (function) using the multiprocessing module and unpack the constants and then append to a list. The try: except: is included as some files are missing metadata and the preprocess_fit function fails and I'd like a nan value to be appended to the list when this occurs.

Issues: 1) Pool cannot unpack function 2) If I only return a const1 from function(files) the processes are appended to the list and not the outputs.

Any suggestion would be great.

def preprocess_fit(file):
    #applies a number of pre-processing steps based on file metadata
    #optimizes fit starting with an initial guess for a parameter until RMS 
    #is minimized
    #returns constants from fitting process and final “guess” parameter
    return const1, const2

def function(files):
    for file in files:
        const1, const2 = preprocess_fit(file)
    return const1, const2

if __name__ == '__main__':
    files = glob.glob("largedata\*.txt")
    p = Pool(24)
    c1 = []
    c2 = []
    import numpy as np
    try:
        const1, const2 = p.map(function, files)
        c1.append(const1)
        c2.append(const2)
    except:
        c1.append(np.nan)
        c2.append(np.nan)
    p.close()
    p.join()

回答1:

When your function is returning multiple items, you will get a list of result-tuples from your pool.map() call. const1 would need all first items in these tuples, const2 all second items in these tuples. That's a job for the zip builtin-function, which returns an iterator that aggregates elements from each of the iterables passed as arguments.

You have to unpack the list so the result-tuples are the arguments for the zip function. Then unpack the iterator by assigning to multiple variables:

const1, const2 = zip(*pool.map(function, files))

来源：https://stackoverflow.com/questions/54575163/how-to-unpack-results-from-pool-map