Overflow in exp in scipy/numpy in Python?

Question

What does the following error:

Warning: overflow encountered in exp

in scipy/numpy using Python generally mean? I\'m computing a ratio in log

Answer 1

When you need to deal with exponential, you quickly go into under/over flow since the function grows so quickly. A typical case is statistics, where summing exponentials of various amplitude is quite common. Since the numbers are very big/smalls, one generally takes the log to stay in a "reasonable" range, the so-called log domain:

exp(-a) + exp(-b) -> log(exp(-a) + exp(-b))

Problems still arise because exp(-a) will still underflows up. For example, exp(-1000) is already below the smallest number you can represent as a double. So for example:

log(exp(-1000) + exp(-1000))

gives -inf (log (0 + 0)), even though you can expect something like -1000 by hand (-1000 + log(2)). The function logsumexp does it better, by extracting the max of the number set, and taking it out of the log:

log(exp(a) + exp(b)) = m + log(exp(a-m) + exp(b-m))

It does not avoid underflow totally (if a and b are vastly different for example), but it avoids most precision issues in the final result