How to set timeout for urlfetch in Google App Engine?

Question

I\'m trying to have Django (on top of GAE) fetch data from another web service. I\'m often hit with error like this:

ApplicationError: 2 timed out Req

Answer 1

You can set it using the deadline argument of the fetch function. From the docs:

The deadline can be up to a maximum of 60 seconds for request handlers and 10 minutes for tasks queue and cron job handlers. If deadline is None, the deadline is set to 5 seconds.

---

Edit: looks like this has changed now. From here:

You can set a deadline for a request, the most amount of time the service will wait for a response. By default, the deadline for a fetch is 5 seconds. You can adjust the default deadline for requests using the urlfetch.set_default_fetch_deadline() function.

And this page lists the default timeout values:

Currently, there are several errors named DeadlineExceededError for the Python runtime:

• google.appengine.runtime.DeadlineExceededError: raised if the overall request times out, typically after 60 seconds, or 10 minutes for task queue requests.
• google.appengine.runtime.apiproxy_errors.DeadlineExceededError: raised if an RPC exceeded its deadline. This is typically 5 seconds, but it is settable for some APIs using the 'deadline' option.
• google.appengine.api.urlfetch_errors.DeadlineExceededError: raised if the URLFetch times out.

Answer 2

It seems short but you have to know that the timeout of a request on GAE is around 30 seconds. As you probably need to do some operations on the response of your urlfetch, there's no need to have a timeout more than 10 seconds I think.

Answer 3

For Go, you might want to try below code.

// createClient is urlfetch.Client with Deadline
func createClient(context appengine.Context, t time.Duration) *http.Client {
    return &http.Client{
        Transport: &urlfetch.Transport{
            Context:  context,
            Deadline: t,
        },
    }
}

Here is how to use it.

// urlfetch
client := createClient(c, time.Second*60)

Answer 4

Seeing as this is a Python question, I thought I'd provide a Python answer for anyone who comes across this problem.

Just import urlfetch and then define a deadline before doing anything else in your code:

from google.appengine.api import urlfetch

urlfetch.set_default_fetch_deadline(60)