Spark UI on AWS EMR

Question

I am running a AWS EMR cluster with Spark (1.3.1) installed via the EMR console dropdown. Spark is current and processing data but I am trying to find which port has been assign

Answer 1

Just run the following command:

ssh -i /your-path/aws.pem -N -L 20888:ip-172-31-42-70.your-region.compute.internal:20888 hadoop@ec2-xxx.compute.amazonaws.com.cn

There are 3 places you need to change:

1. your .pem file
2. your internal master node IP
3. your public DNS domain.

Finally, on the Yarn UI you can click your Spark Application Tracking URL, then just replace the url:

"http://your-internal-ip:20888/proxy/application_1558059200084_0002/" 

->

"http://localhost:20888/proxy/application_1558059200084_0002/"

It worked for EMR 5.x