google-cloud-data-transfer

问题 Closed. This question is not reproducible or was caused by typos. It is not currently accepting answers. Want to improve this question? Update the question so it's on-topic for Stack Overflow. Closed 2 months ago . Improve this question I have access to a project within BigQuery. I'm looking to create a partitioned table by ingestion time, partitioned by day, then set up a BigQuery Data Transfers process that brings avro files in from multiple directories within a Google Cloud Storage Bucket.

问题 Closed. This question is not reproducible or was caused by typos. It is not currently accepting answers. Want to improve this question? Update the question so it's on-topic for Stack Overflow. Closed 2 months ago . Improve this question I have access to a project within BigQuery. I'm looking to create a partitioned table by ingestion time, partitioned by day, then set up a BigQuery Data Transfers process that brings avro files in from multiple directories within a Google Cloud Storage Bucket.

来源： https://stackoverflow.com/questions/63138865/bigquery-data-transfer-from-s3-intermittent-success

问题 I have about 54 000 files in my GCP bucket. When I try to schedule a big query data transfer job to move files from GCP bucket to big query, I am getting the following error: Error code 9 : Transfer Run limits exceeded. Max size: 15.00 TB. Max file count: 10000. Found: size = 267065994 B (0.00 TB) ; file count = 54824. I thought the max file count was 10 million. 回答1: I think that BigQuery transfer service lists all the files matching the wildcard and then use the list to load them. So it

问题 I have about 54 000 files in my GCP bucket. When I try to schedule a big query data transfer job to move files from GCP bucket to big query, I am getting the following error: Error code 9 : Transfer Run limits exceeded. Max size: 15.00 TB. Max file count: 10000. Found: size = 267065994 B (0.00 TB) ; file count = 54824. I thought the max file count was 10 million. 回答1: I think that BigQuery transfer service lists all the files matching the wildcard and then use the list to load them. So it