Spark Failure : Caused by: org.apache.spark.shuffle.FetchFailedException: Too large frame: 5454002341

Question

I am generating a hierarchy for a table determining the parent child.

Below is the configuration used, even after getting the error with regards to the too large fra

Answer 1

This means that size of your dataset partitions is enormous. You need to repartition your dataset to more partitions.

you can do this using,

df.repartition(n)

Here, n is dependent on the size of your dataset.