How to specify which java version to use in spark-submit command?

Question

I want to run a spark streaming application on a yarn cluster on a remote server. The default java version is 1.7 but i want to use 1.8 for my application which is also there in

Answer 1

JAVA_HOME was not enough in our case, the driver was running in java 8, but I discovered later that Spark workers in YARN were launched using java 7 (hadoop nodes have both java version installed).

I had to add spark.executorEnv.JAVA_HOME=/usr/java/ in spark-defaults.conf. Note that you can provide it in command line with --conf.

See http://spark.apache.org/docs/latest/configuration.html#runtime-environment