Create and replace BigQuery tables

问题

How do I create and replace an existing BigQuery table? I use datalab to define BigQuery queries and write the results to BigQuery tables.

The most efficient way I found to do this is:

%%bq query --name helloWorld
Select * from someTable

Followed by

%%bq execute --table schemaName.destination_table --query helloWorld

However I have to manually drop the table each time

From the command line I can execute something like:

bq query --destination_table [PROJECT_ID]:[DATASET].[TABLE] --replace '[QUERY]'

Is there some similar functionality in python/Datalab?

回答1:

Use the --mode or -m parameter. For example:

%%bq execute --table schemaName.destination_table --query helloWorld --mode overwrite

See more options by running %%bq execute --help:

usage: %bq execute [-h] [-nc] [-b BILLING] [-m {create,append,overwrite}] [-l]
                   -q QUERY [-t TABLE] [--to-dataframe]
                   [--dataframe-start-row DATAFRAME_START_ROW]
                   [--dataframe-max-rows DATAFRAME_MAX_ROWS] [-v]

Execute a BigQuery SQL query and optionally send the results to a named table.
The cell can optionally contain arguments for expanding variables in the
query.

optional arguments:
  -h, --help            show this help message and exit
  -nc, --nocache        Don't use previously cached results
  -b BILLING, --billing BILLING
                        BigQuery billing tier
  -m {create,append,overwrite}, --mode {create,append,overwrite}
                        The table creation mode
  -l, --large           Whether to allow large results
  -q QUERY, --query QUERY
                        The name of query to run
  -t TABLE, --table TABLE
                        Target table name
  --to-dataframe        Convert the result into a dataframe
  --dataframe-start-row DATAFRAME_START_ROW
                        Row of the table to start the dataframe export
  --dataframe-max-rows DATAFRAME_MAX_ROWS
                        Upper limit on number of rows to export to the
                        dataframe
  -v, --verbose         Show the expanded SQL that is being executed

来源：https://stackoverflow.com/questions/48980886/create-and-replace-bigquery-tables