google-cloud-python

Google Cloud Storage + Python : Any way to list obj in certain folder in GCS?

由 岁酱吖の 提交于 2019-12-30 06:09:40
问题 I'm going to write a Python program to check if a file is in certain folder of my Google Cloud Storage, the basic idea is to get the list of all objects in a folder, a file name list , then check if the file abc.txt is in the file name list . Now the problem is, it looks Google only provide the one way to get obj list , which is uri.get_bucket() , see below code which is from https://developers.google.com/storage/docs/gspythonlibrary#listing-objects uri = boto.storage_uri(DOGS_BUCKET, GOOGLE

Google Cloud Storage + Python : Any way to list obj in certain folder in GCS?

由 巧了我就是萌 提交于 2019-12-30 06:08:32
问题 I'm going to write a Python program to check if a file is in certain folder of my Google Cloud Storage, the basic idea is to get the list of all objects in a folder, a file name list , then check if the file abc.txt is in the file name list . Now the problem is, it looks Google only provide the one way to get obj list , which is uri.get_bucket() , see below code which is from https://developers.google.com/storage/docs/gspythonlibrary#listing-objects uri = boto.storage_uri(DOGS_BUCKET, GOOGLE

Python Google Cloud Natural Language API returns Bad Gateway error with status 502

由 我只是一个虾纸丫 提交于 2019-12-25 00:36:52
问题 I'm working on a project using Python(3.6) and Django(2) in which I need to process all text files from a directory using Google's Natural Language API python's Client. At the moment, there are 30 text files in my folder for testing purpose, the programs perform sentiment_analysis on 28 files but when reached 29th file it returns an error like below: raise HttpError(resp, content, uri=self.uri) googleapiclient.errors.HttpError: https://language.googleapis.com/v1beta2/documents

Cannot get gcloud to work with Python and Pycharm

由 大兔子大兔子 提交于 2019-12-24 07:07:28
问题 I am trying to connect to the Google App Engine Datastore from my local machine. I have spent all day digging in to this without any luck. I have tried the approach here (as well as alot of other suggestions from SO such as Using gcloud-python in GAE and Unable to run dev_appserver.py with gcloud): How to access a remote datastore when running dev_appserver.py? I first installed gcloud based on this description from google: https://cloud.google.com/appengine/docs/python/tools/using-libraries

Invalid region error when using google-cloud-python API to access Dataproc

由 江枫思渺然 提交于 2019-12-11 02:56:34
问题 I am trying to create a cluster in Dataproc using google-cloud-python library, however, when setting region = 'us-central1' I get below exception: google.api_core.exceptions.InvalidArgument: 400 Region 'us-central1' is invalid. Please see https://cloud.google.com/dataproc/docs/concepts/regional-endpoints for additional information on regional endpoints Code (based on example): #!/usr/bin/python from google.cloud import dataproc_v1 client = dataproc_v1.ClusterControllerClient() project_id =

No module named 'google.cloud._helpers' on GCE

由 社会主义新天地 提交于 2019-12-11 01:32:31
问题 I get an error when importing google.cloud.bigquery on GCE. It did not happen in an environment that is not GCE. I executed update with pip . Do you know why? user1@sv1:~$ python Python 3.6.0 |Continuum Analytics, Inc.| (default, Dec 23 2016,12:22:00) [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from google.cloud import bigquery Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/user1

Permission to Google Cloud Storage via service account in Python

由 為{幸葍}努か 提交于 2019-12-10 16:06:29
问题 I am trying to get a service account to create blobs in Google Cloud Storage from within a Python script, but I am having issues with the credentials. 1) I create the service account for my project and then download the key file in json: "home/user/.config/gcloud/service_admin.json" 2) I give the service account the necessary credentials (via gcloud in a subprocess) roles/viewer, roles/storage.admin, roles/resourcemanager.projectCreator, roles/billing.user Then I would like to access a bucket

Google Cloud Storage: How to Delete a folder (recursively) in Python

由 匆匆过客 提交于 2019-12-05 07:56:55
I am trying to delete a folder in GCS and its all content (including sub-directories) with its Python library. Also I understand GCS doesn't really have folders (but prefix?) but I am wondering how I can do that? I tested this code: from google.cloud import storage def delete_blob(bucket_name, blob_name): """Deletes a blob from the bucket.""" storage_client = storage.Client() bucket = storage_client.get_bucket(bucket_name) blob = bucket.blob(blob_name) blob.delete() delete_blob('mybucket', 'top_folder/sub_folder/test.txt') delete_blob('mybucket', 'top_folder/sub_folder/') The first call to

Efficiently write a Pandas dataframe to Google BigQuery

由 瘦欲@ 提交于 2019-12-03 09:42:59
问题 I'm trying to upload a pandas.DataFrame to google big query using the pandas.DataFrame.to_gbq() function documented here. The problem is that to_gbq() takes 2.3 minutes while uploading directly to Google Cloud Storage GUI takes less than a minute. I'm planing to upload a bunch of dataframes (~32) each one with a similar size, so i want to know what its the faster alternative. This is the script that i'm using: dataframe.to_gbq('my_dataset.my_table', 'my_project_id', chunksize=None, # i've

Efficiently write a Pandas dataframe to Google BigQuery

由 不问归期 提交于 2019-12-03 00:07:33
I'm trying to upload a pandas.DataFrame to google big query using the pandas.DataFrame.to_gbq() function documented here . The problem is that to_gbq() takes 2.3 minutes while uploading directly to Google Cloud Storage GUI takes less than a minute. I'm planing to upload a bunch of dataframes (~32) each one with a similar size, so i want to know what its the faster alternative. This is the script that i'm using: dataframe.to_gbq('my_dataset.my_table', 'my_project_id', chunksize=None, # i've tryed with several chunksizes, it runs faster when is one big chunk (at least for me) if_exists='append',
订阅 google-cloud-python