When running with master 'yarn' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment

Question

I am trying to run Spark using yarn and I am running into this error:

Exception in thread \"main\" java.lang.Exception: When running with master \'yarn\' either

Answer 1

While running spark using Yarn, you need to add following line in to spark-env.sh

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

Note: check $HADOOP_HOME/etc/hadoop is correct one in your environment. And spark-env.sh contains export of HADOOP_HOME as well.