Using XCom to Load Schema in Airflow with GoogleCloudStorageToBigQueryOperator

问题

I have an XCom associated with the Task ID database_schema stored in Airflow that is the JSON schema for a dataset sales_table that I want to load into BigQuery.

The data for the BigQuery dataset sales_table comes from a CSV file retailcustomer_data.csv stored in Google Cloud Storage. The operator for loading the data from GCS to BigQuery is as follows:

gcs_to_bigquery = GoogleCloudStorageToBigQueryOperator(task_id = 'gcs_to_bigquery', bucket = bucket, source_objects = ['retailcustomer_data.csv'], destination_project_dataset_table = dataset_table_name, schema_fields = "{{task_instance.xcom_pull(task_ids='database_schema')}}", write_disposition = 'WRITE_TRUNCATE', bigquery_conn_id = bq_connection, google_cloud_storage_conn_id = gcs_connection, dag = dag)

When the above operator runs as part of the DAG, I am getting the following error message for the gcs_to_bigquery operator. Does anyone know why the XCom associated with Task ID database_schema is not being loaded into schema_fields of the gcs_to_bigquery operator? And how does one fix this issue?

googleapiclient.errors.HttpError: <HttpError 400 when requesting https://bigquery.googleapis.com/bigquery/v2/projects/city_retail_project/jobs?alt=json returned "Invalid value at 'job.configuration.load.schema.fields' (type.googleapis.com/google.cloud.bigquery.v2.TableFieldSchema), "{{task_instance.xcom_pull(task_ids='database_schema')}}"">

回答1:

• The field schema_fields of GoogleCloudStorageToBigQueryOperator is NOT included in template_fields
• So what you are trying will NOT work

...

Citing gotchas from Gtoonstra

Not all parameters in operators are templated, so you cannot use Jinja templates everywhere. The Jinja templates only work for those fields in operators where it’s listed in the template_fields list inside the source file, like:

template_fields = ('audit_key', 'cycle_dtm')

---

Possible workarounds

• Use schema_object field instead
• Extend the operator and define your custom templated fields / logic

来源：https://stackoverflow.com/questions/57931893/using-xcom-to-load-schema-in-airflow-with-googlecloudstoragetobigqueryoperator