triggering_frequency can only be used with FILE_LOADS method of writing to BigQuery

问题

Unable to set triggering_frequency for Dataflow Streaming job.

transformed | 'Write' >> beam.io.WriteToBigQuery(
          known_args.target_table,
          schema=schema,
          create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
          write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND,
          method=bigquery.WriteToBigQuery.Method.FILE_LOADS,
          triggering_frequency=5
          )

Error: triggering_frequency can only be used with FILE_LOADS method of writing to BigQuery

回答1:

This is a bug. The WriteToBigQuery transform has a couple implementations - and the one that supports file loads in streaming is fairly new, thus it is masked behind an experiment.

So: To activate this, please pass use_beam_bq_sink as an experiment, like so: --experiments use_beam_bq_sink.

回答2:

We are still seeing this issue some 9 months later however. Is the fix still to use the experiments=use_beam_bq_sink (because it looks to not be working on our end just now)..

来源：https://stackoverflow.com/questions/57723619/triggering-frequency-can-only-be-used-with-file-loads-method-of-writing-to-bigqu