Google DataFlow Cannot read and write in different locations (Python SDK v0.5.5)

Question

I\'m writing a very basic DataFlow pipeline using the Python SDK v0.5.5. The pipeline uses a BigQuerySource with a query passed in, which is querying BigQuery tables from da

Answer 1

Thanks for reporting this issue. I assume you are using DirectRunner. We changed the implementation of BigQuery read transform for DirectRunner to create a temporary dataset (for SDK versions 0.5.1 and later) to support large datasets. Seems like we are not setting the region correctly here. We'll look into fixing this.

This issue should not occur if you use DataflowRunner which creates temporary datasets in the correct region.