Spark - How to run a standalone cluster locally

Question

Is there the possibility to run the Spark standalone cluster locally on just one machine (which is basically different from just developing jobs locally (i.e., lo

Answer 1

More simply,

./sbin/start-all.sh

On your local machine there will be master and one worker launched.

./bin/spark-submit  \
--class org.apache.spark.examples.SparkPi   \
--master spark://localhost:7077  \
examples/jars/spark-examples_2.12-3.0.1.jar 10000

A sample application is submitted. For monitoring via Web UI:

Master UI:      http://localhost:8080
Worker UI:      http://localhost:8081
Application UI: http://localhost:4040