How to load multiple files (same schema) into a table in BigQuery?

Question

I have a folder of csv files with the same schema that I want to load into a bigquery table.

Is there an option to give folder path as the input to BQ command to lo

Answer 1

Note that

You can use only one wildcard for objects (filenames) within your bucket. The wildcard can appear inside the object name or at the end of the object name. Appending a wildcard to the bucket name is unsupported.

so something like gs://my_bucket/some/*/files* is not supported.

Source: https://cloud.google.com/bigquery/docs/loading-data-cloud-storage#load-wildcards

Answer 2

If using cloud storage is an option, you can put them all in a common prefix in a bucket and use a wildcard e.g. gs://my_bucket/some/path/files* to specify a single load job with multiple inputs quickly.

Answer 3

The files can be in subdirectories, if you want to recursively include all CSV:

bq load --source_format=CSV \
dataset_name.table_name \
"gs://my_bucket/folder/*.csv"

This puts a wildcard on intermediate path and filename. (ex. * expands to subfolder/folder2/filename)