All example concurrent.futures code is failing with “BrokenProcessPool”

Question

I am trying to get a basic understanding of this before I create the actual application I need. I recently moved over from 2.7 to 3.3.

A direct copy-paste of this co

Answer 1

This was my fault, for two reasons:

1. The code was un-guarded, i.e no if __name__
2. The strange looking Traceback was because the file was not saved. Never caused me an issue before, but did in this case.

Correcting both of those fixed the error.

Final test code:

import concurrent.futures

nums = [1,2,3,4,5,6,7,8,9,10]

def f(x):
    return x * x
def main():
    # Make sure the map and function are working
    print([val for val in map(f, nums)])

    # Test to make sure concurrent map is working
    with concurrent.futures.ProcessPoolExecutor() as executor:
        print([val for val in executor.map(f, nums)])

if __name__ == '__main__':
    main()

Output, as expected:

[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

Answer 2

Under Windows, it is important to protect the main loop of code to avoid recursive spawning of subprocesses when using processpoolexecutor or any other parallel code which spawns new processes.

Basically, all your code which creates new processes must be under if __name__ == '__main__': , for the same reason you cannot execute it in interpreter.