How to Auth to Google Cloud using Service Account in Python?

Question

Im trying to make a project that will upload google storage json file to BigQuery (just automate something that is done manually now).

And i\'d like to use \'service

Answer 1

I usually set this variable in the python script itself, something like:

import os
from google.cloud.bigquery.client import Client

os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = 'path_to_json_file'
bq_client = Client()

Answer 2

If you have implemented more fine-grained control over service accounts permissions and you have an app that needs to use several of them (say one for Pub/Sub, one for storage), then you would have to set the GOOGLE_APPLICATION_CREDENTIALS environment variable before creating each client.

Instead, you can load your credentials separately and pass them to their appropriate clients like so:

import json

from google.cloud import storage
from google.oauth2 import service_account

project_id = 'my-test-project'

with open('/path/to/my/service_keys/storage_service.json') as source:
    info = json.load(source)

storage_credentials = service_account.Credentials.from_service_account_info(info)

storage_client = storage.Client(project=project_id, credentials=storage_credentials)

Just make sure in your IAM console that the account has the right permissions to perform the operations you need it to do, but luckily in that case the error messages are really informative.

Answer 3

This is an old question however I want to add that you must create a new service account and not use an old one. A recent Google Cloud Next presentation on security stated that there is no guarantee that the default service account will exist in future, and it could be removed at any time (or its available permissions changed), so none of your applications should depend on it. Also I've found that there are potential issues with authentication when using the default service account and creating a new one is more likely to allow the control you need to successfully authenticate.

Refer to the following YouTube presentation from 11mins 10s in:

https://youtu.be/ZQHoC0cR6Qw?t=670

Answer 4

Are you running the script in the same command line session as the one you set your environment variable in using export? If not, you might want to look into setting it for your user or system (see this question for more info).

Another option that might make things even easier and takes care of these things automatically is using the gcloud cli tool. If you look at the second option here under the How the Application Default Credentials work, you can find info on how to use it to manage the credentials for you (gcloud auth login and gcloud auth application-default login)