integrate() in R gives terribly wrong answer

Question

I was trying to integrate the following function from -infinity to infinity. The answer should be 0.2, but R gives a ridiculously small number. What\'s wrong?

Answer 1

While in general the advice in ?integrate to explicitly specify +/-Inf as the limits is valid, it may be wrong in special cases. This is one of them.

> integrate(f, 20, 30)
0.2 with absolute error < 1.9e-06

The basic problem seems to be that your function is not smooth, in that its derivative is discontinuous at x=25. This may be fooling the algorithm, in particular its use of Wynn's epsilon method to speed up convergence. Basically there's no real substitute to knowing what your function is like, and how its behaviour could cause problems. As pointed out in the answers here, R isn't a symbolic mathematical solver so you do have to exercise more care when trying to get numerical results.

Answer 2

I'll need a bit longer to explain this fully, and hopefully other users will add to this wiki.

From ?integrate, the abs.tol argument is defined as

absolute accuracy requested.

And further down is the following note:

When integrating over infinite intervals do so explicitly, rather than just using a large number as the endpoint. This increases the chance of a correct answer – any function whose integral over an infinite interval is finite must be near zero for most of that interval.

So if you want absolute accuracy as opposed to relative accuracy (which is defined as the result from .Machine$double.eps^0.25) then you can do

> integrate(f, Inf, -Inf, abs.tol = 0L)
0.2 with absolute error < 8.4e-06

The default argument for abs.tol is passed from rel.tol, which is .Machine$double.eps^0.25

Let's see what goes on "inside" a bit.

ifoo<-integrate(f,-Inf,Inf,abs.tol=1e-20)
5.275825e-21 with absolute error < 9.8e-21
str(ifoo)
List of 5
 $ value       : num 5.28e-21
 $ abs.error   : num 9.81e-21
 $ subdivisions: int 3
 $ message     : chr "OK"
 $ call        : language integrate(f = f, lower = -Inf, upper = Inf, abs.tol = 1e-20)
 - attr(*, "class")= chr "integrate"

ifoo<-integrate(f,-Inf,Inf,abs.tol=1e-40)
0.2 with absolute error < 8.4e-06
str(ifoo)
List of 5
 $ value       : num 0.2
 $ abs.error   : num 8.36e-06
 $ subdivisions: int 21
 $ message     : chr "OK"
 $ call        : language integrate(f = f, lower = -Inf, upper = Inf, abs.tol = 1e-40)
 - attr(*, "class")= chr "integrate"

Notice the sudden jump in the number of subdivisions. In general, more subdivisions means better accuracy, which after all is the point of Calculus: reduce the subdivision width to nothing to get the exact answer. My guess is that, with a large(ish) abs.tol, it only takes a few subdivisions for the calculated value to agree with some 'estimated tolerance error' , but when the required tolerance gets small enough, more subdivisions are "added."

Edit: with thanks to Hong Ooi, who actually looked at the integrand in question. :-) . Because this function has a cusp at x==25, i.e. a discontinuity in the derivative, the optimization algorithm likely gets "misled" about convergence. Oddly enough, by taking advantage of the fact that this integrand goes to near-zero very quickly, the result is better when not integrating out to +/-Inf . In fact:

Rgames> integrate(f,20,30)
0.2 with absolute error < 1.9e-06
Rgames> integrate(f,22,27)
0.2 with absolute error < 8.3e-07
Rgames> integrate(f,0,50)
0.2 with absolute error < 7.8e-05