Pyspark Dataframe - Partition by overwrite issue

Question

I am new to Pyspark. I have a created a pyspark script that runs more than once in a month

Sample script:

df = spark.sql("""

SELECT col1,