Uncaught fatal error from thread [default-akka.actor.default-dispatcher-5] shutting down ActorSystem

问题

In my application, I am retriving data from cassandra and providing the rest api using akka spray. It is working fine when I am running through IDE. But when I am running through Spark submit locally on my machine got error as given below

[ERROR] [04/05/2016 10:43:38.687] [default-akka.actor.default-dispatcher-5] [ActorSystem(default)] Uncaught fatal error from thread [default-akka.actor.default-dispatcher-5] shutting down ActorSystem [default] java.lang.NoSuchMethodError: org.apache.spark.sql.cassandra.CassandraSQLContext$$anon$1.DDLStrategy()Lorg/apache/spark/sql/execution/SparkStrategies$DDLStrategy$; at org.apache.spark.sql.cassandra.CassandraSQLContext$$anon$1.(CassandraSQLContext.scala:89) at org.apache.spark.sql.cassandra.CassandraSQLContext.(CassandraSQLContext.scala:85)

My build.sbt as given below.

scalaVersion := "2.10.5"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.4.0"

libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.4.0"

libraryDependencies += "org.apache.spark" % "spark-sql_2.10" % "1.4.0"

libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector" %  "1.4.0" withSources() withJavadoc()


resolvers ++= Seq(
  "Akka Repository" at "http://repo.akka.io/releases/")

resolvers ++= Seq("Typesafe Repository" at "http://repo.typesafe.com/typesafe/releases/",
  "Spray Repository"    at "http://repo.spray.io")


libraryDependencies +=
  "com.typesafe.akka" %% "akka-actor" % "2.3.14"


libraryDependencies ++= {
  val sprayVersion      = "1.3.2"
  Seq(
    "io.spray"          %% "spray-can"       % sprayVersion,
    "io.spray"          %% "spray-routing"   % sprayVersion,
    "io.spray"          %% "spray-json"      % sprayVersion

  )
}

Please let me know what mistake I have done.Thanks in advance. All your suggestion are valueable.

回答1:

You're probably submitting a JAR that contains only your code. You have to add spark-cassandra-connector to class path (for spark-submit) by using --jars parameter (see Spark submit's advanced dependency management)

Or, you can try building a fat JAR (which will include the dependency) and submit it as you're doing so far. You can build a fat JAR from SBT using sbt-assembly

来源：https://stackoverflow.com/questions/36418770/uncaught-fatal-error-from-thread-default-akka-actor-default-dispatcher-5-shutt

标签

apache-spark

akka

spray