Load data from bucket google cloud

问题

Here is a function to load data from google cloud bucket.

action_dataset_folder_path = 'action-data-set'
zip_path = 'actions.zip'
url='http://console.cloud.google.com/storage/browser/actions'

class LoadProgress(tqdm):
    last_block = 0

    def hook(self, block_num=1, block_size=1, total_size=None):
        self.total = total_size
        self.update((block_num - self.last_block) * block_size)
        self.last_block = block_num

if not isfile(zip_path):
    with LoadProgress(unit='B', unit_scale=True, miniters=1, desc='actions-Dataset') as pbar:
        urlretrieve(
            url,
            zip_path,
            pbar.hook)
if not isdir(action_dataset_folder_path):
    with tarfile.open(zip_path) as tar:
        tar.extractall()
        tar.close()
print('All done ...!')

The file is downloaded as empty file with 73.7KB! I did not understand! It seems everything is good.

回答1:

Here is the code from google cloud site:python-code

    from gcloud import storage
    def download_blob(bucket_name, source_blob_name, destination_file_name):
        """Downloads a blob from the bucket."""
        storage_client = storage.Client()
        bucket = storage_client.get_bucket(bucket_name)
        blob = bucket.blob(source_blob_name)

        blob.download_to_filename(destination_file_name)

        print('Blob {} downloaded to {}.'.format(
            source_blob_name,
            destination_file_name))
download_blob("datset","actions", "dataset")

回答2:

You can retrieve data from Google Cloud Storage by using a GET request. In Python you could do this with the Requests library.

First you need to retrieve an auth code (you can test this using OAuth 2.0 Playground)

Then you could use something like this to retrieve the data (object):

import requests

authCode = YOUR_AUTH_CODE
auth = "Bearer " + authCode
myHeaders = {"Authorization": auth}
r = requests.get('https://www.googleapis.com/storage/v1/b/BUCKET_NAME/o/OBJECT_NAME', headers=myHeaders)

print r.text
print r.status_code

来源：https://stackoverflow.com/questions/50383125/load-data-from-bucket-google-cloud