NoClassDefFoundError while using scopt OptionParser with Spark

问题

I am using Apache Spark version 1.2.1 and Scala version 2.10.4. I am trying to get the example MovieLensALS working. However, I am running into errors with scopt library which is a requirement in the code. Any help would be appreciated. My build.sbt is as follows:

name := "Movie Recommender System"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.2.1"

libraryDependencies += "org.apache.spark" %% "spark-graphx" % "1.2.1"

libraryDependencies += "org.apache.spark"  % "spark-mllib_2.10" % "1.2.1"

libraryDependencies += "com.github.scopt" %% "scopt" % "3.2.0"

resolvers += Resolver.sonatypeRepo("public")

and the errors I am getting are the following:

   Exception in thread "main" java.lang.NoClassDefFoundError: scopt/OptionParser
    at MovieLensALS.main(MovieLensALS.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

    Caused by: java.lang.ClassNotFoundException: scopt.OptionParser
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
    ... 8 more
    Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

On running sbt assembly to build the jar, I receive the following errors:

[error] Not a valid command: assembly
[error] Not a valid project ID: assembly
[error] Expected ':' (if selecting a configuration)
[error] Not a valid key: assembly
[error] assembly
[error]         ^

Edit: As per Justin Piphony's suggestion the solution that was listed in sbt's GitHub page helped fix this error. Basically creating a file assembly.sbt in the project/ directory and adding the line addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.13.0")

Note that the version should be added according to the version in use.

回答1:

You need to package scopt in your jar. sbt doesn't do this by default. To create this fat jar, you need to use sbt-assembly

回答2:

If you are using maven to package your spark project, you need to add maven-assembly-plugin plugin which helps to package dependencies:

<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.5</version>
<configuration>
  <descriptorRefs>
    <descriptorRef>jar-with-dependencies</descriptorRef>
  </descriptorRefs>
</configuration>
<executions>
  <execution>
    <id>make-assembly</id>
    <!-- this is used for inheritance merges -->
    <phase>package</phase>
    <!-- bind to the packaging phase -->
    <goals>
      <goal>single</goal>
    </goals>
  </execution>
</executions>

来源：https://stackoverflow.com/questions/29324314/noclassdeffounderror-while-using-scopt-optionparser-with-spark

标签

scala

apache-spark

noclassdeffounderror

scopt