Using non-default service account in Google Cloud dataproc

问题

I'd like to create a dataproc cluster that runs under a non-default service account. The following works for a compute instance:

gcloud compute instances create instance-1 --machine-type "n1-standard-1" --zone "europe-west1-b" --scopes xxxxxxxx@yyyyyyyy.iam.gserviceaccount.com="https://www.googleapis.com/auth/cloud-platform"

But the same --scopes argument fails when creating a dataproc instance:

gcloud dataproc clusters create --zone "europe-west1-b" --scopes xxxxxxxx@yyyyyyyy.iam.gserviceaccount.com="https://www.googleapis.com/auth/cloud-platform" testdataproc12345

ERROR: (gcloud.dataproc.clusters.create) Invalid service account scope: 'xxxxxxxxx@yyyyyyyy.iam.gserviceaccount.com=https://www.googleapis.com/auth/cloud-platform'

Is it possible to run dataproc under a non-default service account?

回答1:

Unfortunately, at the moment there's no way to specify your custom service accounts using the normal "scopes and metadata"-mediated auth setup. This is a known feature request, however, so it should become available in a future Dataproc update.

In the meantime, even though you can't disable the existence of the "storage read/write" scope with the default GCE service account when using Dataproc, you can make the Hadoop side use a particular service account via keyfiles by using the "Create Key" option under the IAM & Admin > Service accounts page to obtain a JSON keyfile for your service account, and then do two things:

Add the following property at cluster creation time:

--properties core:fs.gs.auth.service.account.json.keyfile=/etc/hadoop/conf/my-service-account.json

Use an init action which copies your JSON keyfile to your nodes; note that this still means your JSON keyfile must be accessible to the GCE default service account as a reader, and anyone who has access to the GCS location of your JSON keyfile also has the ability to now act on behalf of that service account, so you still need to keep your project secure as necessary.
```
#!/bin/bash
# Save this somewhere as gs://somepath/my-keyfile-setup.sh

gsutil cp gs://path/to/your/json/file/in/gcs/my=service-account.json \
    /etc/hadoop/conf/my-service-account.json
```
And then apply that init action:
```
gcloud dataproc clusters create --initialization-actions gs://somepath/my-keyfile-setup.sh ...
```

来源：https://stackoverflow.com/questions/38688545/using-non-default-service-account-in-google-cloud-dataproc

标签

google-cloud-platform

google-cloud-dataproc