Scala fat jar dependency issue while Job submit

Question

I have written simple kafka stream using Scala. It is working good in local. I have taken fat jar and submitted in scala cluster. I am getting class not found error after submit

Answer 1

You are missing the kafka dependency:

"org.apache.kafka" %% "kafka" % "0.10.1.0"

Add that an you will be all good.

I have my build.sbt setup like this:

"org.apache.kafka" %% "kafka"                      % "0.10.1.0",
"org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.2.0",
"org.apache.spark" %% "spark-streaming"            % "2.2.0" % "provided"