issue Running Spark Job on Yarn Cluster

Question

I want to run my spark Job in Hadoop YARN cluster mode, and I am using the following command:

spark-submit --master yarn-cluster 
             --dri         


        

Answer 1

For me exit code issue solved by placing hive-site.xml in spark/conf directory.

Answer 2

The command looks correct.

What I've come across is that the "exit code 15" normally indicates a TableNotFound Exception. That usually means there's an error in the code you're submitting.

You can check this by visiting the tracking URL.

Answer 3

It can mean a lot of things, for us, we get the similar error message because of unsupported Java class version, and we fixed the problem by deleting the referenced Java class in our project.

Use this command to see the detailed error message:

yarn logs -applicationId application_1424284032717_0066

Answer 4

You should remove ".setMaster("local")" in the code.

Answer 5

Remove the line "spark.master":"local[*]" in the spark configuration file if you are running the spark jobs under cluster.

Suppose run on the local pc, include it.

Mani