How to read and write Map from/to parquet file in Java or Scala?

Question

Looking for a concise example on how to read and write Map from/to parquet file in Java or Scala?

Here is expected structure, usin

Answer 1

I doubt there is a solution to this readily available. When you talk about Maps, its still possible to create a AvroSchema out of it provided the values of the maps is a primitive type, or a complexType which inturn has primitive type fields.

In your case,

• If you have a Map => which will create schema with values of map being int.
• If you have a Map,
  • a. CustomObject has fields int, float, char ... (i.e. any primitive type) the schema generation will be valid and can then be used to successfully convert to parquet.
  • b. CustomObject has fields which are non primitive, the schema generated will be malformed and the resulting ParquetWritter will fail.

To resolve this issue, you can try to convert your object into a JsonObject and then use the Apache Spark libraries to convert it to Parquet.