Python Memory Error while iterating to a big range

Question

total = 0
x = 2**32
for i in range(x):
    total = total + i;
print(total) 

I am getting a MemoryError<

Answer 1

For a brute-force approach, try this:

x = sum(i for i in xrange(2**32))

The above will be more efficient, as it uses xrange to generate the numbers lazily, and it also uses a generator expression with sum() to avoid generating temporary data that gets discarded immediately.

But that will still take some time, as 2**32 is a big number. The smart way to solve this problem is to use a formula, as suggested by @DeepSpace:

n = 2**32 - 1
x = (n * (n + 1)) / 2

Answer 2

This is what happens when you try to create a list that contains the first 2³² non-negative integers (I'm using Python 2.7.11 on a Windows 10 system):

>>> for i in range(2**32): pass
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: range() result has too many items

One might expect that if the problem is to have such a large number of items simultaneously in memory, the solution could be that of handling one item at a time through a generator... but it is not:

>>> for i in xrange(2**32): pass
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: Python int too large to convert to C long

The documentation of xrange() built-in function explains why this error occurs:

CPython implementation detail: xrange() is intended to be simple and fast. Implementations may impose restrictions to achieve this. The C implementation of Python restricts all arguments to native C longs (“short” Python integers), and also requires that the number of elements fit in a native C long.

The issue is that 2³² cannot be passed as an input argument to xrange because that number is greater than the maximum "short" integer in Python. Try this to convince yourself:

>>> import sys
>>> sys.maxint       # 2**31 - 1
2147483647
>>> sys.maxint + 1   # 2**31 is automatically converted to "long" int
2147483648L
>>> 2**31
2147483648L

You could use nested for loops if you need to repeatedly perform a computation more than 2³¹ times (2³⁴ times in the following example):

>>> loops = 0
>>> for i in xrange(2**4):
...    for j in xrange(2**30):
...        # do stuff
...        loops += 1
...
>>> loops
17179869184L
>>> 2**34
17179869184L

The code above is a rather naïve workaround. A while loop seems to be a much more adequate solution:

>>> loops = 0
>>> while loops < 2**34:
...    # do stuff
...    loops += 1
...
>>> loops
17179869184L

Answer 3

1. range creates a list in memory. Use xrange to get a generator object which gives you one number in a time.

2. There are better ways to sum a range of numbers from 1 to n, for example (n(n+1))/2.