Deadline exceeded while posting multiple cards to timelines with a video attached

Question

first of all i'm using python over gae and i'm trying to push a card with a video attachment to all the users that visited my glassware. This is the code:

        #posting video
        media_link = util.get_full_url(self, '/static/video/man_on_the_moon.mp4')
        resp = urlfetch.fetch(media_link, deadline=2000)
        media_video = MediaIoBaseUpload(io.BytesIO(resp.content), mimetype='video/vnd.google-glass.stream-url',
                                            resumable=True)

        users = Credentials.all()
        for user in users:
            creds = StorageByKeyName(Credentials, user.key().name(), 'credentials').get()
            mirror_service = util.create_service('mirror', 'v1', creds)
            timeline_item = {'text': 'video mode v3'}
            mirror_service.timeline().insert(body=timeline_item, media_body=media_video).execute()

On my local machine this code is working flawless but deploying it on gae generates this:

: HTTPException: Deadline exceeded while waiting for HTTP response from URL: https://www.googleapis.com/upload/mirror/v1/timeline?uploadType=resumable&alt=json&upload_id=AEnB2UqeMjtTAaB7wWw-8yuVoaBdxaD5-mkFx2TJo7PynEXmVCkPLlDFo0l_1t8du_JetszdgmHXF9d-VuD8N0XmwXQVBMynow

Any idea on how to solve this? As far as i understood the timeline().insert uploads the media_body each and every time for each user i post the card to. This seems to me quite resource consuming. Right?

Thanks again guys

this is the full stack:

    Deadline exceeded while waiting for HTTP response from URL: https://www.googleapis.com/upload/mirror/v1/timeline?uploadType=multipart&alt=json
Traceback (most recent call last):
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1511, in __call__
    rv = self.handle_exception(request, response, e)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__
    rv = self.router.dispatch(request, response)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
    return route.handler_adapter(request, response)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__
    return handler.dispatch()
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 547, in dispatch
    return self.handle_exception(e, self.app.debug)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
    return method(*args, **kwargs)
  File "/base/data/home/apps/s~watchup-stage/1.369179617747477845/main_handler.py", line 340, in post
    mirror_service.timeline().insert(body=timeline_item, media_body=media_video).execute()
  File "lib/oauth2client/util.py", line 128, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "lib/apiclient/http.py", line 676, in execute
    body=self.body, headers=self.headers)
  File "lib/oauth2client/util.py", line 128, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "lib/oauth2client/client.py", line 490, in new_request
    redirections, connection_type)
  File "lib/httplib2/__init__.py", line 1570, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "lib/httplib2/__init__.py", line 1317, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "lib/httplib2/__init__.py", line 1286, in _conn_request
    response = conn.getresponse()
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/httplib.py", line 500, in getresponse
    raise HTTPException(str(e))
HTTPException: Deadline exceeded while waiting for HTTP response from URL: https://www.googleapis.com/upload/mirror/v1/timeline?uploadType=multipart&alt=json

Answer 1

I too was experiencing this type of issue. In order to solve it, I had to increase the timeout for HTTP requests as well as move to a Queue.

I submitted a pull request to the Google Mirror API Python example, but it got rejected to keep things "simple".

You can checkout my diff at https://github.com/JessieAMorris/mirror-quickstart-python/commit/b67c0ecdf83207b0c3b596173c95a53804b23525.

Basically, the synopsis is:

1. Move processing to a queue
2. import urlfetch and httplib2

3. Increase the timeouts for urlfetch and httplib2 like so:

   urlfetch.set_default_fetch_deadline(45)
   httplib2.Http(timeout=45) 
   

With those changes in place, I don't get timeouts any more.

Answer 2

You have 60 seconds to complete a front-end request on Google App Engine. Locally however, that restriction doesn't seem to stick.

Your only options for longer processing are task queues (10 minute deadline) and backends (indefinite).