pyspark rdd isCheckPointed() is false

Question

I was encountering stackoverflowerrors when I was iteratively adding over 500 columns to my pyspark dataframe. So, I included checkpoints. The checkpoints did not help. So, I cr

Answer 1

The checkpoint method returns a new check-pointed Dataset, it does not modify the current Dataset.

Change

df4.checkpoint(eager=True)

To

df4 = df4.checkpoint(eager=True)