Spark master-machine:7077 not reachable

问题

I have a Spark Spark cluster where the master node is also the worker node. I can't reach the master from the driver-code node, and I get the error:

14:07:10 WARN client.AppClient$ClientEndpoint: Failed to connect to master master-machine:7077

The SparkContext in driver-code node is configured as: SparkConf conf = new SparkConf(true).setMaster(spark:master-machine//:7077);

I can successfully ping master-machine, but I can't successfully telnet master-machine 7077. Meaning the machine is reachable but the port is not.

What could be the issue? I have disabled Ubuntu's ufw firewall for both master node and node where driver code runs (client).

回答1:

Your syntax is a bit off, you have:

setMaster(spark:master-machine//:7077)

You want:

setMaster(spark://master-machine:7077)

From the Spark docs:

Once started, the master will print out a spark://HOST:PORT URL for itself, which you can use to connect workers to it, or pass as the “master” argument to SparkContext. You can also find this URL on the master’s web UI, which is http://localhost:8080 by default.

You can use an IP address in there too, I have run into issues with debian-based installs where I always have to use the IP address but that's a separate issue. An example:

spark.master            spark://5.6.7.8:7077

From a configuration page in Spark docs

来源：https://stackoverflow.com/questions/38557223/spark-master-machine7077-not-reachable