问题
We are doing the data transformations using Google Dataproc and all our data is residing in Dataproc Hive tables. How do i transfer/move this data to BigQuery.
回答1:
Transfer to BigQuery from Hive seems to have a standard pattern:
- dump your Hive into Avro files
- Load those files in BigQuery
See an example here: Migrate hive table to Google BigQuery
As mentioned above, take care about the types compatibility between Hive/Avro/BigQuery.
And for the first time I guess it would not hurt to do some validations by comparing that the tables on both Hive and BigQuery have the same data: https://github.com/bolcom/hive_compared_bq
来源：https://stackoverflow.com/questions/47900265/move-data-from-hive-tables-in-google-dataproc-to-bigquery