spark submit “Service 'Driver' could not bind on port” error

Question

I used the following command to run the spark java example of wordcount:-

time spark-submit --deploy-mode cluster --master spark://192.168.0.7:6066 --class o         


        

Answer 1

This is possibly a duplicate of Spark 1.2.1 standalone cluster mode spark-submit is not working

I have tried the same steps, but able to run the job. Kindly post the full spark-env.sh and spark-defaults if possible.

Answer 2

This issue is related to IP address alone. Error messages in the log file are not informative. check with following 3 steps:

1. check your IP address - can be checked with ifconfig or ip commands. If your service is not a Public service. IP addresses with 192.168 should be good enough. 127.0.0.1 cannot be used if you are planning a cluster.

2. check your environment variable SPARK_MASTER_HOST - check there are no typos in the name of the variable or actual IP address.

   env | grep SPARK_

3. check the port you are planning to use for sparkMaster is free with command netstat. Do not use a port below 1024. For example:

   netstat -a | 9123

After your sparkmaster starts running if you are not able see webui from a different machine, then open the webui port with command iptables.

Answer 3

I had this problem and it is because of changing real IP with my IP in /etc/hosts.