How to spark-submit with main class in jar?

Question

There are a ton of questions about ClassNotFoundException but I haven\'t seen any (yet) that fit this specific case. I am attempting to run the following command:

Answer 1

Why don't you use the path to the jar file so spark-submit (as any other command line tool) could find and use it?

Given the path out/artifacts/TimeSeriesFilter_jar/scala-ts.jar I'd use the following:

spark-submit --class com.stronghold.HelloWorld out/artifacts/TimeSeriesFilter_jar/scala-ts.jar

Please note that you should be in the project's main directory which seems to be /home/[USER]/projects/scala_ts.

Please also note that I removed --master local[*] since that's the default master URL spark-submit uses.

Answer 2

Afraid none of these were the issue. I had previously tried deleting everything in the project and starting over, but that didn't work either. Once it occurred to me start an entirely different project, it worked just fine. Apparently Intellij (of which I am a fan) decided to create a hidden problem somewhere.

Answer 3

Try to add maven-shade plugin and build > run.

Here is the reference, it may help you.

Getting java.lang.ClassNotFoundException while running Spark Application

Answer 4

Looking at your pom file, the jar file you are referencing doesn't match with what you have in the pom file as

  <artifactId>scala-ts</artifactId>
  <version>1.0-SNAPSHOT</version>

The above two line in pom file suggests that your jar file should be scala-ts-1.0-SNAPSHOT.jar but you have used scala-ts.jar only. So I am assuming you are referencing old jar.

Here are few steps you can apply

1. clean the project and package again
2. make sure the jar file name by going to target folder of the project
3. you can give the exact path to the target folder to point to the jar when you apply spark-submit command