How do I access my AppEngine DataStore entities from my Compute Engine VM?

Question

My app is running on App Engine, but I would like to access its NDB DataStore entities from my Compute Engine VM to do some processing and write the results back to the App

Answer 1

The officially recommended way is to use the datastore client libraries; see https://cloud.google.com/datastore/docs/reference/libraries You need to create a service account, or use the standard compute engine service account, give permission to either all APIs, or to datastore to that service account, and create the compute engine instance to be part of that service account. See here for more information. Then you can do something like:

from google.auth import compute_engine                                                                                      
from google.cloud import datastore                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                

datastore_client = datastore.Client(project='yourproject',     
                                    credentials=compute_engine.Credentials())                                                                                                                                                                  
q = datastore_client.query(kind='YourEntity')                                                                                    
q.add_filter('field_name', '=', 'HelloThere') 
print list(q.fetch(1))

Answer 2

Google Cloud Datastore is kind of a standalone version of App Engine's datastore.

Back to your problem, you have two options :

1. Write a web-service to expose your entities from the App Engine app to the Compute Engine VMs. One option is Cloud Endpoints. Cloud Endpoints uses OAuth2 authentication and the Compute Engine VMs are shipped with OAuth2 service accounts that you can use to authenticate to the service.

2. Use the App Engine remote API to access the Datastore. The remote API provides access to the Datastore as if you were in an App Engine instance. But by default the API requires you to provide the password of an App Engine amdin, so you might have to store your password in the Compute Engine VMs, which is not safe.

Answer 3

Aside from options that @David explained, you can also look into Managed VMs: they are a cross between Compute Engine and App Engine - basically a managed Compute Engine instance that has access to App Engine services.

Answer 4

David's solution requires you to use App Engine instance time to make requests, but you can bypass it and make requests directly to Datastore from Compute Engine instance. There is a pretty good tutorial about how to do this. But its not so pretty like ndb.

>>> import googledatastore as datastore
>>> datastore.set_options(dataset='project-id')
>>> req = datastore.BeginTransactionRequest()
>>> datastore.begin_transaction(req)
<datastore.datastore_v1_pb2.BeginTransactionResponse object at ...>