How do I return data from a deferred task in Google App Engine

Question

Original Question

I have a working version of my web application that I am trying to upgrade at the moment, and I\'m running into the issue of having a task which

Answer 1

Normally you can't reply to the original request anymore since the context of that original request dissapears. Maybe, if you return from the request handler without replying and if somehow that doesn't kill the connection from the client and if you are somehow able to persist the handler object so that you can later restore it in another (internal) request and use the restored copy to reply from it to the original request... Kind of a long shot at best.

One option would be to split the operation into a sequence: - a 1st request starting the operation - subsequent one or more polling requests until the operation completes and the result is available

Another approach may be possible if the expensive operation is mainly executing on data available prior to when the operation is invoked. You could re-org the app logic so that partial results are computed as soon as the respective data becomes available, so that when the final operation is requested it only operates on pre-computed partial results. An analogy, if you want, would be Google search requests immediately receiving replies with data from pre-computed indexes instead of waiting for an actual web search to be performed.

Answer 2

Well, first, it's already bad to let users wait for 1 minute until page loads. In general, user-facing HTTP requests should take no more than 1 second. Those 60 seconds that GAE gives -- is already too generous, for critical situations.

I have several suggestions, but I don't know your application to say what you need:

1. Precompute. Load, compute and store lineups value before user request it. For that you can utilize GAE Backend instances, which can run way longer than 60 seconds.
2. Do users really need that much data? Generally, if there's so much data that computer has problems sorting it -- it's already too much to show to user. Probably your users just need to see some small part of it (like top 10 players, or some aggregate statistics). Then improvement of algorithm used in makeLineups() will do the trick.
3. Defer. If you cannot do 1 or 2, then your option is to defer the computation to Task API. For that your frontend should:
4. Enqueue a task using Task Queue: https://cloud.google.com/appengine/docs/python/taskqueue/
   • Open channel to user using Channel API: https://cloud.google.com/appengine/docs/python/channel/
   • Save the channel_id for that user to Datastore.
   • Finish the call. On UI show user a message like "please wait, we're crunching down the numbers".
   • At the same time, GAE backend executes the task you enqueued. The task computes value of makeLineups(). Once done, the task will take channel_id from Datastore and send there the computed value of lineups.
   • User frontend receives the value and makes user happy.
5. Instead of Task API there's new Background Threads that may be easier and better for your case: https://cloud.google.com/appengine/docs/python/modules/#Python_Background_threads Basically, instead of enqueueing a task, you call'd background_thread.BackgroundThread(), the rest stays the same. UPDATE This will work better only with backend modules (basic or manual scaling, not automatic). On Frontend (default) modules, custom threads cannot outlive HTTP request, and hence also limited to 60s.

Let me know if that helps.

Answer 3

I'm not familiar with GAE, but this is a fairly generic question, so I can give you some advice.

Your general idea is correct, so I'm just going to expand on it. The workflow could look like this:

1. You get the request to create the lineups. You create a new entity in the datastore for it. It should contain an ID (you'll need it to retrieve the result later) and a status (PENDING|DONE|FAILED). You can also save the data from the request, if that's useful to you.
2. You defer the computation and return a response right away. The response will contain the ID of the task. When the computation is done, it will save the result of the task in the Datastore and update the status of the task. That result will contain the task ID, so that we can easily find it.
3. Once the frontend receives the ID, it starts polling for the result. Using setTimeout or setInterval you send requests with the task ID to the server (this is a separate endpoint). The server checks the status of the task, and returns the result if it's done (error if failed).
4. The frontend gets the data and stops polling.