Pyspark S3 error: java.lang.NoClassDefFoundError: com/amazonaws/services/s3/model/MultiObjectDeleteException

Question

Been unsuccessful setting a spark cluster that can read AWS s3 files. The software I used are as follows:

1. hadoop-aws-3.2.0.jar
2. aws-java-sdk-1.11.887.jar<

Answer 1

Hadoop 3.2 was built against 1.11.563; stick the full shaded sdk of that specific version in your classpath "aws-java-sdk-bundle" and all should be well.

The SDK has been "fussy" in the past...and upgrade invariably causes surprises. For the curious Qualifying an AWS SDK update. It's probably about time someone does it again.