How do I return data from a deferred task in Google App Engine

Question

Original Question

I have a working version of my web application that I am trying to upgrade at the moment, and I\'m running into the issue of having a task which

Answer 1

Well, first, it's already bad to let users wait for 1 minute until page loads. In general, user-facing HTTP requests should take no more than 1 second. Those 60 seconds that GAE gives -- is already too generous, for critical situations.

I have several suggestions, but I don't know your application to say what you need:

1. Precompute. Load, compute and store lineups value before user request it. For that you can utilize GAE Backend instances, which can run way longer than 60 seconds.
2. Do users really need that much data? Generally, if there's so much data that computer has problems sorting it -- it's already too much to show to user. Probably your users just need to see some small part of it (like top 10 players, or some aggregate statistics). Then improvement of algorithm used in makeLineups() will do the trick.
3. Defer. If you cannot do 1 or 2, then your option is to defer the computation to Task API. For that your frontend should:
4. Enqueue a task using Task Queue: https://cloud.google.com/appengine/docs/python/taskqueue/
   • Open channel to user using Channel API: https://cloud.google.com/appengine/docs/python/channel/
   • Save the channel_id for that user to Datastore.
   • Finish the call. On UI show user a message like "please wait, we're crunching down the numbers".
   • At the same time, GAE backend executes the task you enqueued. The task computes value of makeLineups(). Once done, the task will take channel_id from Datastore and send there the computed value of lineups.
   • User frontend receives the value and makes user happy.
5. Instead of Task API there's new Background Threads that may be easier and better for your case: https://cloud.google.com/appengine/docs/python/modules/#Python_Background_threads Basically, instead of enqueueing a task, you call'd background_thread.BackgroundThread(), the rest stays the same. UPDATE This will work better only with backend modules (basic or manual scaling, not automatic). On Frontend (default) modules, custom threads cannot outlive HTTP request, and hence also limited to 60s.

Let me know if that helps.