Spark Standalone Cluster - Slave not connecting to Master

Question

I am trying to setup a Spark standalone cluster following the official documentation.

My master is on a local vm running ubuntu and I also have one worker running in

Answer 1

Its depends on your spark version, it will need different conf. if your spark version 1.6 add this line to conf/spark-env.shso another machine can connect to master

SPARK_MASTER_IP=your_host_ip

and if your spark version is 2.x add these lines to your conf/spark-env.sh

SPARK_MASTER_HOST=your_host_ip

SPARK_LOCAL_IP=your_host_ip

after adding these lines run spark :

./sbin/spark-all.sh

and if you do right , you can see in <your_host_ip>:8080 that spark master url is:spark://<your_host_ip>:7077

BeCarefule your_host_ip ,shouldnt be localhost and It must be exactly Your host ip that you set in conf/spark-env.sh

after all you can connect another machine to the master with command below:

./sbin/start-slave.sh spark://your_host_ip:7077

Answer 2

I encounter the exact same problem as you and just figure out how to get it to work.

The problem is that your spark master is listening on hostname, in your example spark, which causes the worker on the same host being able to register successfully but failed from another machine with command start-slave.sh spark://spark:7077.

The solution is to make sure the value SPARK_MASTER_IP is specified with ip in file conf/spark-env.sh

    SPARK_MASTER_IP=<your host ip>

on your master node, and start your spark master as normal. You can open your web GUI to make sure your spark master appears as spark://YOUR_HOST_IP:7077 after the start. Then, on another machine with command start-slave.sh spark://<your host ip>:7077 should start and register worker to master successfully.

Hope it would help you

Answer 3

I just launched my own spark cluster with version 2.10. The way I solved my problem is the following:

 ./sbin/start-master.sh -h <your_ip>

Answer 4

All solutions above didn't work for me, but I found one more way to fix problem: Stackoverflow: spark worker not connecting to master

Please check configuration file "spark-env.sh" on your master node. Have you set the SPARK_MASTER_HOST variable to the IP address of the master node? If not try to set it and restart the master and slaves. For example, if your master node's IP is 192.168.0.1, you should have SPARK_MASTER_HOST=192.168.0.1 in there. Note that you don't need to set this variable on your slaves.