Exception in thread “main” java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps

Question

When I run in terminal:

sudo spark-submit --master local --class xxx.xxxx.xxx.xxxx.xxxxxxxxxxxxJob --conf \'spark.driver.extraJavaOptions=-Dconfig.resource=xxx.con         


        

Answer 1

You're mixing scala versions. Spark 2.4.2 doesn't support scala 2.11. Switch to Spark 2.4.0 or replace your libraries with scala 2.12 versions.

https://spark.apache.org/releases/spark-release-2-4-2.html

Note that Scala 2.11 support is deprecated from 2.4.1 onwards. As of 2.4.2, the pre-built convenience binaries are compiled for Scala 2.12. Spark is still cross-published for 2.11 and 2.12 in Maven Central, and can be built for 2.11 from source.