How to create AWS Athena table via Glue crawler when the s3 data store has both json and .gz compressed files?

Question

I have two problems in my intended solution:

1. My S3 store structure is as following:

mainfolder/date=2019-01-01/hour=14/abcd.json
mainfolder/date=2019-         


        

Answer 1

I would suggest skipping using a crawler altogether. In my experience Glue crawlers are not worth the problems they cause. It's easy to create tables with the Glue API, and so is adding partitions. The API is a bit verbose, especially adding partitions, but it's much less pain than trying to make a crawler do what you want it to do.

You can of course also create the table from Athena, that way you can be sure you get tables that work with Athena (otherwise there are some details you need to get right). Adding partitions is also less verbose using SQL through Athena, but slower.

Answer 2

Crawler will not take compressed and uncompressed data together , so it will not work out of box. It is better to write spark job in glue and use spark.read()