java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.reloadExistingConfigurations()V

Question

It looks like I am again stuck on the running a packaged spark app jar using spark submit. Following is my pom file:

Answer 1

The dependencies between hadoop and AWS JDK are very sensitive, and you should stick to using the correct versions that your hadoop dependency version was built with.

The first problem you need to solve is pick one version of Hadoop. I see you're mixing versions 2.8.3 and 2.8.0.

When I look at the dependency tree for org.apache.hadoop:hadoop-aws:2.8.0, I see that it is built against version 1.10.6 of the AWS SDK (same for hadoop-aws:2.8.3).

This is probably what's causing mismatches (you're mixing incompatible versions). So:

• Choose the version of hadoop you want to use
• Include hadoop-aws with the version compatible with your hadoop
• Remove other dependencies, or only include them with versions matching the one compatible with your hadoop version.