Spark throws java.io.IOException: Failed to rename when saving part-xxxxx.gz

Question

New Spark user here. I\'m extracting features from many .tif images stored on AWS S3, each with identifier like 02_R4_C7. I\'m using Spark 2.2.1 and hadoop 2.7.2.

I\'m

Answer 1

The solutions in @Steve Loughran post are great. Just to add a little info to help explaining the issue.

Hadoop-2.7 uses Hadoop Commit Protocol for committing. When Spark saves result to S3, it actually saves temporary result to S3 first and make it visible by renaming it when job succeeds (reason and detail can be found in this great doc). However, S3 is an object store and does not have real "rename"; it copy the data to target object, then delete original object.

S3 is "eventually consistent", which means the delete operation could happen before copy is fully synced. When this happens, the rename would fail.

In my cases, this was only triggered in some chained jobs. I haven't seen this in simple save job.