Overflow Error in Python's numpy.exp function

Question

I want to use numpy.exp like this:

cc = np.array([
    [0.120,0.34,-1234.1]
])

print 1/(1+np.exp(-cc))

But this gives me erro

Answer 1

The largest value representable by a numpy float is 1.7976931348623157e+308, whose logarithm is about 709.782, so there is no way to represent np.exp(1234.1).

In [1]: import numpy as np

In [2]: np.finfo('d').max
Out[2]: 1.7976931348623157e+308

In [3]: np.log(_)
Out[3]: 709.78271289338397

In [4]: np.exp(709)
Out[4]: 8.2184074615549724e+307

In [5]: np.exp(710)
/usr/local/bin/ipython:1: RuntimeWarning: overflow encountered in exp
  #!/usr/local/bin/python3.5
Out[5]: inf

Answer 2

A possible solution is to use the decimal module, which lets you work with arbitrary precision floats. Here is an example where a numpy array of floats with 100 digits precision is used:

import numpy as np
import decimal

# Precision to use
decimal.getcontext().prec = 100

# Original array
cc = np.array(
    [0.120,0.34,-1234.1]
)
# Fails
print(1/(1 + np.exp(-cc)))    

# New array with the specified precision
ccd = np.asarray([decimal.Decimal(el) for el in cc], dtype=object)
# Works!
print(1/(1 + np.exp(-ccd)))

Answer 3

exp(-1234.1) is too small for 32bit or 64bit floating-point numbers. Since it cannot be represented, numpy produces the correct warning.

Using IEEE 754 32bit floating-point numbers, the smallest positive number it can represent is 2^(-149), which is roughly 1e-45.

If you use IEEE 754 64 bit floating-point numbers, the smallest positive number is 2^(-1074) which is roughy 1e-327.

In either case, it cannot represent a number as small as exp(-1234.1) which is about 1e-535.

You should be using the expit function from scipy to compute the sigmoid function. This would give you better precision.

For practical purposes, exp(-1234.1) is a very small number. If rounding to zero makes sense in your use case, numpy produces benign results by rounding it to zero.

Answer 4

As fuglede says, the issue here is that np.float64 can't handle a number as large as exp(1234.1). Try using np.float128 instead:

>>> cc = np.array([[0.120,0.34,-1234.1]], dtype=np.float128)
>>> cc
array([[ 0.12,  0.34, -1234.1]], dtype=float128)
>>> 1 / (1 + np.exp(-cc))
array([[ 0.52996405,  0.58419052,  1.0893812e-536]], dtype=float128)

Note however, that there are certain quirks with using extended precision. It may not work on Windows; you don't actually get the full 128 bits of precision; and you might lose the precision whenever the number passes through pure python. You can read more about the details here.

For most practical purposes, you can probably approximate 1 / (1 + <a large number>) to zero. That is to say, just ignore the warning and move on. Numpy takes care of the approximation for you (when using np.float64):

>>> 1 / (1 + np.exp(-cc))
/usr/local/bin/ipython3:1: RuntimeWarning: overflow encountered in exp
  #!/usr/local/bin/python3.4
array([[ 0.52996405,  0.58419052,  0.        ]])

If you want to suppress the warning, you could use scipy.special.expit, as suggested by WarrenWeckesser in a comment to the question:

>>> from scipy.special import expit
>>> expit(cc)
array([[ 0.52996405,  0.58419052,  0.        ]])