Deadline Exceeded Error even with a TaskQueue running on a Backend

Question

I\'m having some issues with Deadline Exceeded error. Basically I\'m doing some webscrapping in an URL using Mechanize. So when trying to perfo

Answer 1

No request that involves getting data via HTTP can take more then 60 seconds on app engine.

The 10 minute limit refers to the tasks themselves - they can run for up to 10 minutes.

So GAE might not be the best choice here as you can only use it's provided versions of urlfetch etc, if your requests are going to take longer then 60 seconds on average anyway.

You can set a deadline for a request, the most amount of time the service will wait for a response. By default, the deadline for a fetch is 5 seconds. The maximum deadline is 60 seconds for HTTP requests and 10 minutes for task queue and cron job requests.

https://developers.google.com/appengine/docs/python/urlfetch/

So a task can run for up to 10 minutes and a url fetch for (max) 60 seconds. It does not matter where you perform the urlfetch operation from, a front or backend, the limit is the same.