Exporting data to GCS from BigQuery - Split file size control

Question

I am currently exporting data from Bigquery to GCS buckets. I am doing this programmatically using the following query:

query_request = bigquery_service.jobs()

         


        

Answer 1

If you specify multiple URI patterns, the data will be sharded between them. So if you used, say, 28 URI patterns, each shard would be about half a GB. You'd end up with second files of size zero for each pattern, as this is really meant for MR jobs, but its one way to accomplish what you want.

More info here (see the Multiple Wildcard URIs section): Exporting Data From BigQuery