google-cloud-storage

问题 I have an XCom associated with the Task ID database_schema stored in Airflow that is the JSON schema for a dataset sales_table that I want to load into BigQuery. The data for the BigQuery dataset sales_table comes from a CSV file retailcustomer_data.csv stored in Google Cloud Storage. The operator for loading the data from GCS to BigQuery is as follows: gcs_to_bigquery = GoogleCloudStorageToBigQueryOperator(task_id = 'gcs_to_bigquery', bucket = bucket, source_objects = ['retailcustomer_data

问题 I am listening to data from pub-sub using streaming data in dataflow. Then I need to upload to storage, process the data and upload it to bigquery. here is my code: public class BotPipline { public static void main(String[] args) { DataflowPipelineOptions options = PipelineOptionsFactory.as(DataflowPipelineOptions.class); options.setRunner(BlockingDataflowPipelineRunner.class); options.setProject(MY_PROJECT); options.setStagingLocation(MY_STAGING_LOCATION); options.setStreaming(true);

问题 I am listening to data from pub-sub using streaming data in dataflow. Then I need to upload to storage, process the data and upload it to bigquery. here is my code: public class BotPipline { public static void main(String[] args) { DataflowPipelineOptions options = PipelineOptionsFactory.as(DataflowPipelineOptions.class); options.setRunner(BlockingDataflowPipelineRunner.class); options.setProject(MY_PROJECT); options.setStagingLocation(MY_STAGING_LOCATION); options.setStreaming(true);

问题 I am listening to data from pub-sub using streaming data in dataflow. Then I need to upload to storage, process the data and upload it to bigquery. here is my code: public class BotPipline { public static void main(String[] args) { DataflowPipelineOptions options = PipelineOptionsFactory.as(DataflowPipelineOptions.class); options.setRunner(BlockingDataflowPipelineRunner.class); options.setProject(MY_PROJECT); options.setStagingLocation(MY_STAGING_LOCATION); options.setStreaming(true);

问题 I tried the bq command line tool, but it there seems to be no way to write the results directly to GCS. The only way I can see is working is: query -> table -> export query -> console -> local CSV file -> upload to GCS I'm looking for a way to directly write the results to GCS. 回答1: You cannot write your query results directly to GCS. The easiest way, as you've already pointed out, is to: Write your query results to an output table Export your table to GCS You easily do this in the web UI.

问题 It is okay when dealing with small files. It doesn't work only when I try to upload large files. I'm using Python client. The snippet is: filename='my_csv.csv' storage_client = storage.Client() bucket_name = os.environ["GOOGLE_STORAGE_BUCKET"] bucket = storage_client.get_bucket(bucket_name) blob = bucket.blob("{}".format(filename)) blob.upload_from_filename(filename) # file size is 500 MB The only thing I get as a Traceback is "Killed" and I'm out of python interpreter. Any suggestions are

问题 I'm using google-api-php-client Here's the bit where I upload a .jpg image. $postbody = array("data" => $imgData); $gso = new Google_StorageObject(); $gso->setName($imageName); $contentType = 'image/jpg'; $gso->setContentType($contentType); $resp = $objects->insert('bucket-name', $gso, $postbody); By inspecting $gso ContentType is being added but in the cloud console is added with the default application/octet-stream type. Is there another way to set the content type? 回答1: Try this, it worked

问题 I'd like to trigger a Dataflow job when new files are added to a Storage bucket in order to process and add new data into a BigQuery table. I see that Cloud Functions can be triggered by changes in the bucket, but I haven't found a way to start a Dataflow job using the gcloud node.js library. Is there a way to do this using Cloud Functions or is there an alternative way of achieving the desired result (inserting new data to BigQuery when files are added to a Storage bucket)? 回答1: This is

问题 I'm trying to generate download and upload link from the Google Cloud, to view and upload files, using the following code: public class Test { public static void main(String[] args) throws IOException { Storage storage = StorageOptions.newBuilder().setCredentials( ServiceAccountCredentials.fromStream(new FileInputStream("C:/cred/Key.json"))) .build() .getService(); String filePath = "file/path/"; File file = new File(filePath); byte[] bytes = Utilities.fileToByteArray(file); String mimeType =

问题 I'd like to connect securely to SQL Google Cloud Platform using Google Apps Script . However, I'm not sure how to set the parameters _serverSslCertificate, _clientSslCertificate , and clientSslKey of Jdbc.getConnection method. I already downloaded cert and key files. -----BEGIN RSA PRIVATE KEY----- MIIEpAIBAAKCAQEAlj6vYtt6F8rv254sllmvGogSYwzS0JaB6Yezji6Ab+gmZPLc dyfNIg4PTe8YCz45RnJ+8VVklTQ4K/MFzsB6nilNMDSFRCAfRmytQkioEFlgzdL+ ... JOyb/68aCqHHStwlpCUTH40Iqh3kneiQ1be2XigxjllMSYkuH/ebZw== ----