Why do Scala 2.11 and Spark with scallop lead to “java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror”?

Question

I am using Scala 2.11, Spark, and Scallop (https://github.com/scallop/scallop). I used sbt to build an application fat jar without Spark provided dependencies (this is at: <

Answer 1

I was also encountered with the same issue with spark-submit, in my case:

Spark Job was compiled with : Scala 2.10.8

Scala version with which Spark was compiled on the cluster: Scala 2.11.8

To check the Spark version and Scala version on the cluster use "spark-shell" command.

After compiling the Spark Job source with Scala 2.11.8 then submitted the job & it worked !!!.