How to create AWS Athena table via Glue crawler when the s3 data store has both json and .gz compressed files?

Question

I have two problems in my intended solution:

1. My S3 store structure is as following:

mainfolder/date=2019-01-01/hour=14/abcd.json
mainfolder/date=2019-         


        

Answer 1

Crawler will not take compressed and uncompressed data together , so it will not work out of box. It is better to write spark job in glue and use spark.read()