Python Memory Error while iterating to a big range

Question

total = 0
x = 2**32
for i in range(x):
    total = total + i;
print(total) 

I am getting a MemoryError<

Answer 1

This is what happens when you try to create a list that contains the first 2³² non-negative integers (I'm using Python 2.7.11 on a Windows 10 system):

>>> for i in range(2**32): pass
... 
Traceback (most recent call last):
  File "", line 1, in 
OverflowError: range() result has too many items

One might expect that if the problem is to have such a large number of items simultaneously in memory, the solution could be that of handling one item at a time through a generator... but it is not:

>>> for i in xrange(2**32): pass
... 
Traceback (most recent call last):
  File "", line 1, in 
OverflowError: Python int too large to convert to C long

The documentation of xrange() built-in function explains why this error occurs:

CPython implementation detail: xrange() is intended to be simple and fast. Implementations may impose restrictions to achieve this. The C implementation of Python restricts all arguments to native C longs (“short” Python integers), and also requires that the number of elements fit in a native C long.

The issue is that 2³² cannot be passed as an input argument to xrange because that number is greater than the maximum "short" integer in Python. Try this to convince yourself:

>>> import sys
>>> sys.maxint       # 2**31 - 1
2147483647
>>> sys.maxint + 1   # 2**31 is automatically converted to "long" int
2147483648L
>>> 2**31
2147483648L

You could use nested for loops if you need to repeatedly perform a computation more than 2³¹ times (2³⁴ times in the following example):

>>> loops = 0
>>> for i in xrange(2**4):
...    for j in xrange(2**30):
...        # do stuff
...        loops += 1
...
>>> loops
17179869184L
>>> 2**34
17179869184L

The code above is a rather naïve workaround. A while loop seems to be a much more adequate solution:

>>> loops = 0
>>> while loops < 2**34:
...    # do stuff
...    loops += 1
...
>>> loops
17179869184L