Pyspark — AWS EMR: Job aborted due to stage failure: ShuffleMapStage — Failed to connect to ip-XXX-XX-XX-XX.ec2.internal/XXX.XX.XX.XX:XXXXX

Question

I have a monthly data pipeline on AWS EMR that used to run fine. This previous run, we received a much higher load of data than usual. Now when I submit a job I start hittin