NoSuchMethodError while running Spark Streaming job on HDP 2.2

问题

I am trying to run a simple streaming job on HDP 2.2 Sandbox but facing java.lang.NoSuchMethodError error. I am able to run SparkPi example on this machine without an issue.

Following are the versions I am using-

    <kafka.version>0.8.2.0</kafka.version>
    <twitter4j.version>4.0.2</twitter4j.version>
    <spark-version>1.2.1</spark-version>
    <scala.version>2.11</scala.version>

Code Snippet -

val sparkConf = new SparkConf().setAppName("TweetSenseKafkaConsumer").setMaster("yarn-cluster");
val ssc = new StreamingContext(sparkConf, Durations.seconds(5));

Error text from Node Manager UI-

Exception in thread "Driver" scala.MatchError: java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less; (of class java.lang.NoSuchMethodError) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:432) 15/02/12 15:07:23 INFO yarn.ApplicationMaster: Waiting for spark context initialization ... 1 15/02/12 15:07:33 INFO yarn.ApplicationMaster: Waiting for spark context initialization ... 2

Job is accepted in YARN but it never goes into RUNNING status.

I was thinking it is due to Scala version differences. I tried changing POM configuration but still not able to fix the error.

Thank you for your help in advance.

回答1:

Earlier I specified dependency for spark-streaming_2.10 ( Spark compiled with Scala 2.10). I did not specify dependency for Scala compiler itself. It seems Maven automatically pulled 2.11 (Maybe due to some other dependency). When trying to debug this issue, I added a dependency on Scala compiler 2.11. Now after Paul's comment I changed that Scala dependency version to 2.10 and it is working.

来源：https://stackoverflow.com/questions/28487305/nosuchmethoderror-while-running-spark-streaming-job-on-hdp-2-2