Why do Scala 2.11 and Spark with scallop lead to “java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror”?

Question

I am using Scala 2.11, Spark, and Scallop (https://github.com/scallop/scallop). I used sbt to build an application fat jar without Spark provided dependencies (this is at: <

Answer 1

The issue is that you've used incompatible Scala versions, i.e. Spark was compiled with Scala 2.10 and you were trying to use Scala 2.11.

Move everything to Scala 2.10 version and make sure you update your SBT as well.

You may also try to compile Spark sources for Scala 2.11.7 and use it instead.

Answer 2

I was also encountered with the same issue with spark-submit, in my case:

Spark Job was compiled with : Scala 2.10.8

Scala version with which Spark was compiled on the cluster: Scala 2.11.8

To check the Spark version and Scala version on the cluster use "spark-shell" command.

After compiling the Spark Job source with Scala 2.11.8 then submitted the job & it worked !!!.