java.lang.NoClassDefFoundError: org/apache/hadoop/fs/StorageStatistics

Question

I\'m trying to run a simple spark to s3 app from a server but I keep getting the below error because the server has hadoop 2.7.3 installed and it looks like it doesn\'t incl

Answer 1

You can't mix bits of Hadoop and expect things to work. It's not just the close coupling between internal classes in hadoop-common and hadoop-aws, its things like the specific version of the amazon-aws SDK the hadoop-aws module was built it.

Unless you can isolate classpaths, you'll need to roll your POM back to 2.7.3.

Sorry