Python large iterations number fail

Question

I wrote simple monte-carlo π calculation program in Python, using multiprocessing module. It works just fine, but when I pass 1E+10 iterations for each worker, some problem occur, and the result is wrong. I cant understand what is the problem, because everything is fine on 1E+9 iterations!

import sys
from multiprocessing import Pool
from random import random


def calculate_pi(iters):
    """ Worker function """

    points = 0  # points inside circle

    for i in iters:
        x = random()
        y = random()

        if x ** 2 + y ** 2 <= 1:
            points += 1

    return points


if __name__ == "__main__":

    if len(sys.argv) != 3:
        print "Usage: python pi.py workers_number iterations_per_worker"
        exit()

    procs = int(sys.argv[1])
    iters = float(sys.argv[2])  # 1E+8 is cool

    p = Pool(processes=procs)

    total = iters * procs
    total_in = 0

    for points in p.map(calculate_pi, [xrange(int(iters))] * procs):
        total_in += points

    print "Total: ", total, "In: ", total_in
    print "Pi: ", 4.0 * total_in / total

Answer 1

The problem seems to be that multiprocessing has a limit to the largest int it can pass to subprocesses inside an xrange. Here's a quick test:

import sys
from multiprocessing import Pool
def doit(n):
  print n
if __name__ == "__main__":
  procs = int(sys.argv[1])
  iters = int(float(sys.argv[2]))
  p = Pool(processes=procs)
  for points in p.map(doit, [xrange(int(iters))] * procs):
    pass

Now:

$ ./multitest.py 2 1E8
xrange(100000000)
xrange(100000000)
$ ./multitest.py 2 1E9
xrange(1000000000)
xrange(1000000000)
$ ./multitest.py 2 1E10
xrange(1410065408)
xrange(1410065408)

This is part of a more general problem with multiprocessing: It relies on standard Python pickling, with some minor (and not well documented) extensions to pass values. Whenever things go wrong, the first thing to check is that the values are arriving the way you expected.

In fact, you can see this problem by playing with pickle, without even touching multiprocessing (which isn't always the case, because of those minor extensions, but often is):

>>> pickle.dumps(xrange(int(1E9)))
'c__builtin__\nxrange\np0\n(I0\nI1000000000\nI1\ntp1\nRp2\n.'
>>> pickle.dumps(xrange(int(1E10)))
'c__builtin__\nxrange\np0\n(I0\nI1410065408\nI1\ntp1\nRp2\n.'

Even without learning all the details of the pickle protocol, it should be obvious that the I1000000000 in the first case is 1E9 as an int, while the equivalent chunk of the next case is about 1.41E9, not 1E10, as an int. You can experiment

One obvious solution to try is to pass int(iters) instead of xrange(int(iters)), and let calculate_pi create the xrange from its argument. (Note: In some cases an obvious transformation like this can hurt performance, maybe badly. But in this case, it's probably slightly better if anything—a simpler object to pass, and you're parallelizing the xrange construction—and of course the difference is so tiny it probably won't matter. Just make sure to think before blindly transforming.)

And a quick test shows that this now works:

import sys
from multiprocessing import Pool

def doit(n):
  print xrange(n)

if __name__ == "__main__":
    procs = int(sys.argv[1])
    iters = int(float(sys.argv[2]))
    p = Pool(processes=procs)
    for points in p.map(doit, [iters] * procs):
      pass

Then:

$ ./multitest.py 2 1E10
xrange(10000000000)
xrange(10000000000)

However, you will still run into a larger limit:

$ ./multitest.py 2 1E100
OverflowError: Python int too large to convert to C long

Again, it's the same basic problem. One way to solve that is to pass the arg all the way down as a string, and do the int(float(a)) inside the subprocesses.

As a side note: The reason I'm doing iters = int(float(sys.argv[2])) instead of just iters = float(sys.argv[2]) and then using int(iters) later is to avoid accidentally using the float iters value later on (as the OP's version does, in computing total and therefore total_in / total).

And keep in mind that if you get to big enough numbers, you run into the limits of the C double type: 1E23 is typically 99999999999999991611392, not 100000000000000000000000.