Spark UI on AWS EMR

Question

I am running a AWS EMR cluster with Spark (1.3.1) installed via the EMR console dropdown. Spark is current and processing data but I am trying to find which port has been assign

Answer 1

Here is an alternative if you don't want to deal with the browser setup with SOCKS as suggested on the EMR docs.

1. Open a ssh tunnel to the master node with port forwarding to the machine running spark ui

   ssh -i path/to/aws.pem  -L 4040:SPARK_UI_NODE_URL:4040 hadoop@MASTER_URL
   

   MASTER_URL (EMR_DNS in the question) is the URL of the master node that you can get from EMR Management Console page for the cluster

   SPARK_UI_NODE_URL can be seen near the top of the stderr log. The log line will look something like:

   16/04/28 21:24:46 INFO SparkUI: Started SparkUI at http://10.2.5.197:4040
   

2. Point your browser to localhost:4040

Tried this on EMR 4.6 running Spark 2.6.1