Is it possible to load Avro files with Snappy compression into BigQuery?

问题

I know that BigQuery supports Avro file upload and I'm successful in loading Avro file into BigQuery.

Using below command,

java -jar avro-tools-1.7.7.jar fromjson --codec snappy --schema-file SourceSchema.avsc Source.json > Output.snappy.avro

I have generated an Avro file with Snappy compression and trying to load into BigQuery but Load job fails with below errors,

Errors:
file-00000000: The Apache Avro library failed to parse file file-00000000. (error code: invalid)

Is it possible to load Avro files with Snappy compression into BigQuery?

回答1:

BigQuery only supports DEFLATE and Snappy algorithm for Avro data blocks compression, from the docs (https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro)

Compressed Avro files are not supported, but compressed data blocks are. BigQuery supports the DEFLATE and Snappy codecs.

回答2:

Now BigQuery supports Snappy. See: https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro

Compressed Avro files are not supported, but compressed data blocks are. BigQuery supports the DEFLATE and Snappy codecs.

来源：https://stackoverflow.com/questions/39739171/is-it-possible-to-load-avro-files-with-snappy-compression-into-bigquery