Python's multiprocessing and memory

Question

I am using multiprocessing.imap_unordered to perform a computation on a list of values:

def process_parallel(fnc, some_list):
    pool = multiproces         


        

Answer 1

The simplest solution I can think of would be to add a closure to wrap your fnc function which would use a semaphore to control the total number of simultaneous job executions that can execute at one time (I assume the main process/thread would be incrementing the semaphore). The semaphore value could be calculated based on job size and available memory.