Windows: Apache Spark History Server Config

Question

I wanted to use Spark\'s History Server to make use of the logging mechanisms of my Web UI, but I find some difficulty in running this code on my Windows machine.

I

Answer 1

On Windows you'll need to run the .cmd files of Spark not .sh. According to what I saw, there is no .cmd script for Spark history server. So basically it needs to be run manually.

I have followed the history server Linux script and in order to run it manually on Windows you'll need to take the following steps:

• All history server configurations should be set at the spark-defaults.conf file (remove .template suffix) as described below

• You should go to spark config directory and add the spark.history.* configurations to %SPARK_HOME%/conf/spark-defaults.conf. As follows:

  spark.eventLog.enabled true spark.history.fs.logDirectory file:///c:/logs/dir/path

• After configuration is finished run the following command from %SPARK_HOME%

  bin\spark-class.cmd org.apache.spark.deploy.history.HistoryServer

• The output should be something like that:

  16/07/22 18:51:23 INFO Utils: Successfully started service on port 18080. 16/07/22 18:51:23 INFO HistoryServer: Started HistoryServer at http://10.0.240.108:18080 16/07/22 18:52:09 INFO ShutdownHookManager: Shutdown hook called

Hope that it helps! :-)