Split table into multiple tables based on date using bigquery with a single query for partitioning

前端 未结 2 1914
孤城傲影
孤城傲影 2021-01-27 01:34

The original \"why\" of what I want to do is:

Restore a table maintaining its original partitioning instead of it all going into today\'s partition.

What I thoug

2条回答
  •  陌清茗
    陌清茗 (楼主)
    2021-01-27 02:15

    Answering myself here. Another approach I've seen done is to write a script that:

    1. Parses the tablebackup.json file, outputs many files tablebackuppartitionYYYYMMDD.json split on a provided parameter.

    2. Creates a batch script to bq load all the files into the appropriate table partitions.

    The script would need to process row by row or chunks to be able to handle massive backups. And would take some time. The advantage of using this method is it would be generic and usable by an untrained-in-BQ sysadmin.

提交回复
热议问题