Using gcloud-python in GAE

Question

I\'ve got a bunch of little Raspberry Pis running some python code which saves directly to the Datastore (skips GAE) using the gcloud-python datastore package. This works gr

Answer 1

GAE on your Mac can't access Python packages installed in the default place on your Mac. You need to do this:

ln -s /Library/Python/2.7/site-packages/.../gcloud /Users/sheridangray/Projects/city-pulse-web/gcloud

(need to replace ... with the relevant path info)

Answer 2

You can run gcloud-python on App Engine, but it requires a little extra work. Check out a skeleton project I wrote which gets this working.

The main bases to cover are:

Getting the dependencies

In install_gcloud.sh, pip is used to install gcloud and its dependencies inside the application using

pip install --target="application/vendor/" gcloud

(As mentioned in the other answer, local installs don't get uploaded to App Engine on deploy.)

Modifying the dependencies

By pip-installing with --target set, pkg_resources.get_distribution will work as expected (it failed in your stack trace).

In addition pytz by default has too many reads to work well on App Engine, so gae-pytz is used instead. As a result, some pytz imports need to be modified.

In addition, to reduce a Compute Engine check overhead (network overhead), the oauth2client.client module can be modified.

Both of these tweaks can be found in a single commit.

Making Imports Work

The script above puts all dependencies in a directory called vendor/ and appengine_config.py adds this to the import path via Darth Vendor:

import darth
darth.vendor('vendor')

In addition, since the protobuf dependency is also part of the google package (just like all the App Engine imports, e.g. google.appengine.ext.ndb) you need to modify the __path__ associated with that package:

import os
import google

curr_dir = os.path.abspath(os.path.dirname(__file__))
vendor_dir = os.path.join(curr_dir, 'vendor')
google.__path__.append(os.path.join(vendor_dir, 'google'))

CAVEAT (as of January 22, 2015)

Be very aware that using gcloud-python within App Engine will be 3-5x slower than using the native App Engine libraries db or ndb. This is because those use direct RPCs into the App Engine runtime while gcloud-python will use HTTP outside of App Engine to talk to the Cloud Datastore API.

---

NOTE: I updated this after the initial posting, which referenced a previous point in history.