Python multiprocessing: why are large chunksizes slower?

Question

I\'ve been profiling some code using Python\'s multiprocessing module (the \'job\' function just squares the number).

data = range(100000000)
n=4
time1 = tim         


        

Answer 1

About optimal chunksize:

1. Having tons of small chunks would allow the 4 different workers to distribute the load more efficiently, thus smaller chunks would be desirable.
2. In the other hand, context changes related to processes add an overhead everytime a new chunk has to be processed, so less amount of context changes and therefore less chunks are desirable.

As both rules want different aproaches, a point in the middle is the way to go, similar to a supply-demand chart.