Integer overflow in numpy arrays

Question

import numpy as np
a = np.arange(1000000).reshape(1000,1000)
print(a**2)

With this code I get this answer. Why do I get negative values?

Answer 1

On your platform, np.arange returns an array of dtype 'int32' :

In [1]: np.arange(1000000).dtype
Out[1]: dtype('int32')

Each element of the array is a 32-bit integer. Squaring leads to a result which does not fit in 32-bits. The result is cropped to 32-bits and still interpreted as a 32-bit integer, however, which is why you see negative numbers.

Edit: In this case, you can avoid the integer overflow by constructing an array of dtype 'int64' before squaring:

a=np.arange(1000000,dtype='int64').reshape(1000,1000)

Note that the problem you've discovered is an inherent danger when working with numpy. You have to choose your dtypes with care and know before-hand that your code will not lead to arithmetic overflows. For the sake of speed, numpy can not and will not warn you when this occurs.

See http://mail.scipy.org/pipermail/numpy-discussion/2009-April/041691.html for a discussion of this on the numpy mailing list.

Answer 2

A solution to this problem is as follows (taken from here):

...change in class StringConverter._mapper (numpy/lib/_iotools.py) from:

{{{
 _mapper = [(nx.bool_, str2bool, False),
            (nx.integer, int, -1),
            (nx.floating, float, nx.nan),
            (complex, _bytes_to_complex, nx.nan + 0j),
            (nx.string_, bytes, asbytes('???'))]
}}}

to

{{{
 _mapper = [(nx.bool_, str2bool, False),
            (nx.int64, int, -1),
            (nx.floating, float, nx.nan),
            (complex, _bytes_to_complex, nx.nan + 0j),
            (nx.string_, bytes, asbytes('???'))]
 }}}

This solved a similar problem that I had with numpy.genfromtxt for me

Note that the author describes this as a 'temporary' and 'not optimal' solution. However, I have had no side effects using v2.7 (yet?!).

Answer 3

numpy integer types are fixed width and you are seeing the results of integer overflow.

Answer 4

python integers don't have this problem, since they automatically upgrade to python long integers when they overflow.

so if you do manage to overflow the int64's, one solution is to use python int's in the numpy array:

import numpy
a=numpy.arange(1000,dtype=object)
a**20