google-cloud-data-fusion

Possible to modify or delete rows from a table in BigQuery dataset with a Cloud Data Fusion pipeline?

由 十年热恋 提交于 2019-12-24 07:39:08
问题 I have a requirement to build a Data Studio dashboard and to use data from BigQuery dataset. I have imported my data to BQ using Data Fusion from an on-premise MS SQL server, and the requirement is I have to delete the last 5 days of the records and import new updated records for the same time range on top of the records in the BQ dataset... So far I was able to do all the work with the pipeline but when I run the pipeline it does append the data again into the BQ table and I end up with

Getting Null Pointer Exception when mapping SQL Server Database to MySQL Database with MapReduce

由 半腔热情 提交于 2019-12-11 16:54:24
问题 I am new to Cloud Data Fusion and am trying to map tables in a SQL Server Database to a MySQL Database. I have already faced many issues which I managed to solve namely: Fixed permissions for the service account so it could access all the resources it required; Added IP to the allowed connections in my SQL Server; Am using system.profile.properties.dataproc:dataproc.conscrypt.provider.enable = false to prevent SSL bug issue as reported in another question. After this last fix, I am now trying

While running Data fusion pipeline to load csv file from GCS to BigQuery facing some issue regarding data-proc deprovisioning

由 北慕城南 提交于 2019-12-07 06:13:38
问题 I am using Data fusion to create a pipeline which will load CSV data from GCS to BigQuery. When i am doing the preview it's working fine. But when i am deploying the pipeline it's giving me below error. ERROR io.cdap.cdap.internal.provision.task.ProvisioningTask#151-provisioning-service-13 DEPROVISION task failed in REQUESTING_DELETE state for program run program_run:default.gcstobqsample.-SNAPSHOT.workflow.DataPipelineWorkflow.31a8341b-70d6-11e9-9c94-92fdc3807015. com.google.api.gax.rpc

Can't connect Cloud Data Fusion with Google Cloud SQL for PostgreSQL

由 ♀尐吖头ヾ 提交于 2019-12-04 04:05:44
问题 My goal is to read data from Cloud SQL Postgres to BigQuery via a Cloud Data Fusion pipeline. For this, I set up a Cloud Data Fusion instance and assigned the following two permissions to the service account: (see https://cloud.google.com/data-fusion/docs/how-to/create-instance#setting_up_permissions) Cloud SQL Client Cloud Data Fusion API Service Agent As a next step I connected myself to the Cloud Data Fusion Instance, and navigated to Wrangler -> Add Connection -> Database -> Google Cloud

Can't connect Cloud Data Fusion with Google Cloud SQL for PostgreSQL

由 岁酱吖の 提交于 2019-12-01 19:56:20
My goal is to read data from Cloud SQL Postgres to BigQuery via a Cloud Data Fusion pipeline. For this, I set up a Cloud Data Fusion instance and assigned the following two permissions to the service account: (see https://cloud.google.com/data-fusion/docs/how-to/create-instance#setting_up_permissions ) Cloud SQL Client Cloud Data Fusion API Service Agent As a next step I connected myself to the Cloud Data Fusion Instance, and navigated to Wrangler -> Add Connection -> Database -> Google Cloud SQL for PostgreSQL . As the driver I uploaded the postgres-socket-factory-1.0.13-jar-with-dependencies
订阅 google-cloud-data-fusion