Spark Kafka Streaming Issue

Question

I am using maven

i have added the following dependencies

    
      org.apache.spark&         


        

Answer 1

For future reference, if you get a ClassNotFoundException, if you search for "org.apache.spark..." you will be taken to the maven page where it will tell you the dependency you are missing in your pom file. It will also give you the code to put in your pom file.

Answer 2

I meet the same problem, I solved it by build the jar with dependencies.

1. remove "sc.addJar()" in your code.

2. add the code below to pom.xml

   <build>
       <sourceDirectory>src/main/java</sourceDirectory>
       <testSourceDirectory>src/test/java</testSourceDirectory>
       <plugins>
         <!--
                      Bind the maven-assembly-plugin to the package phase
           this will create a jar file without the storm dependencies
           suitable for deployment to a cluster.
          -->
         <plugin>
           <artifactId>maven-assembly-plugin</artifactId>
           <configuration>
             <descriptorRefs>
               <descriptorRef>jar-with-dependencies</descriptorRef>
             </descriptorRefs>
             <archive>
               <manifest>
                 <mainClass></mainClass>
               </manifest>
             </archive>
           </configuration>
           <executions>
             <execution>
               <id>make-assembly</id>
               <phase>package</phase>
               <goals>
                 <goal>single</goal>
               </goals>
             </execution>
           </executions>
         </plugin>
       </plugins>
   </build>    
   

3. mvn package

4. submit the "example-jar-with-dependencies.jar"