Invalid region error when using google-cloud-python API to access Dataproc

问题

I am trying to create a cluster in Dataproc using google-cloud-python library, however, when setting region = 'us-central1' I get below exception:

google.api_core.exceptions.InvalidArgument: 400 Region 'us-central1' is invalid.
Please see https://cloud.google.com/dataproc/docs/concepts/regional-endpoints
for additional information on regional endpoints

Code (based on example):

#!/usr/bin/python

from google.cloud import dataproc_v1

client = dataproc_v1.ClusterControllerClient()

project_id = 'my-project'
region = 'us-central1'
cluster = {...}

response = client.create_cluster(project_id, region, cluster)

回答1:

Dataproc uses region field for routing REST requests, however, the field is not used in gRPC clients (hence the error).

Only the global multiregion can be accessed through the default endpoint. To use a regional endpoint such as us-central1, you have to configure the endpoint to address on the client's transport.

The Dataproc regional endpoints follow this pattern: <region>-dataproc.googleapis.com:443. The region field should be set to the same value as the region in the endpoint.

Example:

#!/usr/bin/python

from google.cloud import dataproc_v1
from google.cloud.dataproc_v1.gapic.transports import cluster_controller_grpc_transport

transport = cluster_controller_grpc_transport.ClusterControllerGrpcTransport(
    address='us-central1-dataproc.googleapis.com:443')
client = dataproc_v1.ClusterControllerClient(transport)

project_id = 'my-project'
region = 'us-central1'
cluster = {...}

response = client.create_cluster(project_id, region, cluster)

回答2:

Similarly using the google-cloud-java client:

ClusterControllerSettings settings =
     ClusterControllerSettings.newBuilder()
        .setEndpoint("us-central1-dataproc.googleapis.com:443")
        .build();
 try (ClusterControllerClient clusterControllerClient = ClusterControllerClient.create(settings)) {
   String projectId = "my-project";
   String region = "us-central1";
   Cluster cluster = Cluster.newBuilder().build();
   Cluster response =
       clusterControllerClient.createClusterAsync(projectId, region, cluster).get();
 }

回答3:

As for now, the recommended way to change the default API endpoint is to use client_options:

client_options (Union[dict, google.api_core.client_options.ClientOptions]) – Client options used to set user options on the client. API Endpoint should be set through client_options.

Here's an example with loading credentials from json file (Python 3.6+ syntax with f-string):

from google.cloud.dataproc_v1 import ClusterControllerClient


client = ClusterControllerClient.from_service_account_file(
             service_account_json_path,
             client_options={'api_endpoint': f'{your_region}-dataproc.googleapis.com:443'})

来源：https://stackoverflow.com/questions/53401219/invalid-region-error-when-using-google-cloud-python-api-to-access-dataproc

标签

google-cloud-dataproc

google-cloud-python