HTTPS get using “requests” module in Google App Engine fails

Question

I want to use the requests module in Google App Engine Python Standard Runtime Environment.

Quote from official Google Cloud docs:

Yo

Answer 1

This was a pain to setup for python 2.7 in GAE standard. It involved using app engine's beta version of their version of the python ssl library and a few other odds and ends.

I'm sure you'll face some differences for python3. This is were the key bits for me to get it working:

requests 2.18.2

requests_toolbelt 0.7.0

in appengine_config.py do this:

from requests_toolbelt.adapters import appengine as requests_toolbelt_appengine

# Use the App Engine Requests adapter. This makes sure that Requests uses
# URLFetch.
requests_toolbelt_appengine.monkeypatch()

in app.yaml have these:

env_variables:
  GAE_USE_SOCKETS_HTTPLIB : 'true'


libraries:
- name: ssl
  version: "2.7.11"
- name: pycrypto
  version: "2.6"

FUTHERMORE, this got it working for me in production, but not on my localhost. In addition to a local lib directory with all my third party libraries, I had to setup an additional directory called localhost_libs and do this in appengine_config.py as well:

vendor.add('lib')

if os.environ.get('SERVER_SOFTWARE', '').startswith('Development'):
    vendor.add('localhost_libs')

where I have pycrypto

Also, for a long time everyone had to do this too (eventually something changed in dev_appserver that stopped this from working): "ImportError: No module named _ssl" with dev_appserver.py from Google App Engine