I am reading json by spark there is nothing special just:
spark.read.option(\'compression\', \'gzip\').option(\'dropFieldIfAllNull\', True)\\ .json(source_final)