spark throws java.lang.NoClassDefFoundError: kafka/common/TopicAndPartition

问题

when I use spark-submit command in Cloudera Yarn environment, I got this kind of exception:

java.lang.NoClassDefFoundError: kafka/common/TopicAndPartition
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.getDeclaredMethods(Class.java:1975)
    at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.com$fasterxml$jackson$module$scala$introspect$BeanIntrospector$$listMethods$1(BeanIntrospector.scala:93)
    at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.findMethod$1(BeanIntrospector.scala:99)
    at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.com$fasterxml$jackson$module$scala$introspect$BeanIntrospector$$findGetter$1(BeanIntrospector.scala:124)
    at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$$anonfun$3$$anonfun$apply$5.apply(BeanIntrospector.scala:177)
    at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$$anonfun$3$$anonfun$apply$5.apply(BeanIntrospector.scala:173)
    at scala.collection.TraversableLike$WithFilter$$anonfun$map$2.apply(TraversableLike.scala:722)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
    at scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:721)
    at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$$anonfun$3.apply(BeanIntrospector.scala:173)
    at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$$anonfun$3.apply(BeanIntrospector.scala:172)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
    at scala.collection.immutable.List.foreach(List.scala:318)
...

The spark-submit command is like:

spark-submit --master yarn-cluster \
        --num-executors $2 \
        --executor-cores $3 \
        --class "APP" \
        --deploy-mode cluster \
        --properties-file $1 \
        --files $HDFS_PATH/log4j.properties,$HDFS_PATH/metrics.properties \
        --conf spark.metrics.conf=metrics.properties \
        APP.jar

note that, TopicAndPartition.class is in shaded APP.jar.

回答1:

Please try adding the Kafka jar using the --jars option as shown in the example below:

spark-submit --master yarn-cluster \
    --num-executors $2 \
    --executor-cores $3 \
    --class "APP" \
    --deploy-mode cluster \
    --properties-file $1 \
    --jars /path/to/kafka.jar
    --files $HDFS_PATH/log4j.properties,$HDFS_PATH/metrics.properties \
    --conf spark.metrics.conf=metrics.properties \
    APP.jar

回答2:

After using some methods, it turns out that the issue is caused because version incompatibility. As @user1050619 said, make sure the version of kafka, spark, zookeeper and scala are compatible with each other.

来源：https://stackoverflow.com/questions/39284887/spark-throws-java-lang-noclassdeffounderror-kafka-common-topicandpartition

标签

apache-spark

apache-kafka

spark-streaming