Apache Spark Exception in thread “main” java.lang.NoClassDefFoundError: scala/collection/GenTraversableOnce$class

Question

Scala version:2.11.7(had to upgrade the scala verison to enable case clasess to accept more than 22 parameters.) Spark version:1.6.1 PFB pom.xml

Getting below error

Answer 1

In the POM you have Scala version 2.11.7 but later on in the dependencies you are declaring Spark deps compiled against 2.10:

<artifactId>spark-streaming_2.10</artifactId>
<artifactId>spark-core_2.10</artifactId>
<artifactId>spark-sql_2.10</artifactId>

You have to change it to:

<artifactId>spark-streaming_2.11</artifactId>
<artifactId>spark-core_2.11</artifactId>
<artifactId>spark-sql_2.11</artifactId>

Answer 2

For scala 2.12.0, you should add the following dependency to your POM.xml file, and that's it.

<!-- https://mvnrepository.com/artifact/org.scala-lang/scala-library -->
<dependency>
    <groupId>org.scala-lang</groupId>
    <artifactId>scala-library</artifactId>
    <version>2.12.0-M1</version>
</dependency>