I have been facing issue while processing dataframe in pyspark. The dataframe has following columns. The data is increasing per day basis.The code was working fine for few month