Spark worker can not connect to Master

Question

While starting the worker node I get the following error :

Spark Command: /usr/lib/jvm/default-java/bin/java -cp /home/ubuntu/spark-1.5.1-bin-hadoop2.6/sbin/         


        

Answer 1

I'm afraid your hostname may be invalid to Spark, and you hava to change your spark-env.sh.

You can set the variable SPARK_MASTER_IP to be the real ip of master, instead of its hostname. e.g.

export SPARK_MASTER_IP=1.70.44.5

INSTEAD OF

export SPARK_MASTER_IP=ip-1-70-44-5