Spark throws java.io.IOException: Failed to rename when saving part-xxxxx.gz

Question

New Spark user here. I\'m extracting features from many .tif images stored on AWS S3, each with identifier like 02_R4_C7. I\'m using Spark 2.2.1 and hadoop 2.7.2.

I\'m

Answer 1

It's not safe to use S3 as a direct destination of work without a "consistency layer" (Consistent EMR, or from the Apache Hadoop project itself, S3Guard), or a Special output committer designed explicitly for work with S3 (Hadoop 3.1+ "the S3A committers"). Rename is where things fail, as listing inconsistency means that the scan for files to copy may miss data, or find deleted files which it can't rename. Your stack trace looks exactly how I'd expect this to surface: job commits failing apparently at random.

Rather than go into the details, here's a video of Ryan Blue on the topic

Workaround: write to your local cluster FS then use distcp to upload to S3.

PS: for Hadoop 2.7+, switch to the s3a:// connector. It has exactly the same consistency problem without S3Guard enabled, but better performance.