How to query BigQuery programmatically from Python without end-user interaction?

Question

This question seems like it should be so simple to answer, but after days of research and several dead ends, I can\'t seem to get query results out of BigQuery with

Answer 1

I had the same issue. It is possibly because you do not have the requisite permissions in the Project for the Service Account key you are using.

Answer 2

If you login locally with gcloud:

gcloud auth application-default login

Then a credentials file will be stored in ~/.config/gcloud/, which can be loaded with:

from oauth2client.client import GoogleCredentials
from apiclient.discovery import build

credentials = GoogleCredentials.get_application_default()
service = build('bigquery', 'v2')

Or you can just load the BQ security key file directly with:

from google.cloud import bigquery
client = bigquery.Client.from_service_account_json(path_to_key.json)

Answer 3

Sorry this is being so challenging to find info on. You're looking for what's called Service Accounts which are documented in our Authorizing Access to the BigQuery API using OAuth 2.0 guide.

Here's an example, using the Python client library, though you'll want to look at the referenced documentation for info on acquiring the appropriate credentials:

import httplib2

from apiclient.discovery import build
from oauth2client.client import SignedJwtAssertionCredentials

# REPLACE WITH YOUR Project ID
PROJECT_NUMBER = 'XXXXXXXXXXX'
# REPLACE WITH THE SERVICE ACCOUNT EMAIL FROM GOOGLE DEV CONSOLE
SERVICE_ACCOUNT_EMAIL = 'XXXXX@developer.gserviceaccount.com'

# OBTAIN THE KEY FROM THE GOOGLE APIs CONSOLE
# More instructions here: http://goo.gl/w0YA0
f = file('key.p12', 'rb')
key = f.read()
f.close()

credentials = SignedJwtAssertionCredentials(
    SERVICE_ACCOUNT_EMAIL,
    key,
    scope='https://www.googleapis.com/auth/bigquery')

http = httplib2.Http()
http = credentials.authorize(http)

service = build('bigquery', 'v2')
datasets = service.datasets()
response = datasets.list(projectId=PROJECT_NUMBER).execute(http)

print 'Dataset list:'
for dataset in response['datasets']:
  print '%s' % dataset['datasetReference']['datasetId']