I am using spark 1.4 and trying to save as orcfile with compression snappy but it saves as zlib

问题

here is my code:

val df=hiveContext.write.format("orc").options("orc.compression","SNAPPY").save(
"xyz")

but file is saved as ZLIB.

回答1:

You could try adding the extra conf "spark.io.compression.codec=snappy" to spark-shell / spark-submit:

spark-shell --conf spark.io.compression.codec=snappy #rest of your command..

Also, for writing to ORC format (assuming you are in Spark >= 1.5) you can use:

myDf.orc("/some/path")

The "orc" method is exactly like doing '.format("orc").save("/some/path")'.

来源：https://stackoverflow.com/questions/33464990/i-am-using-spark-1-4-and-trying-to-save-as-orcfile-with-compression-snappy-but-i