Windows: Apache Spark History Server Config

Question

I wanted to use Spark\'s History Server to make use of the logging mechanisms of my Web UI, but I find some difficulty in running this code on my Windows machine.

I

Answer 1

On Windows you'll need to run the .cmd files of Spark not .sh. According to what I saw, there is no .cmd script for Spark history server. So basically it needs to be run manually.

I have followed the history server Linux script and in order to run it manually on Windows you'll need to take the following steps:

• All history server configurations should be set at the spark-defaults.conf file (remove .template suffix) as described below

• You should go to spark config directory and add the spark.history.* configurations to %SPARK_HOME%/conf/spark-defaults.conf. As follows:

  spark.eventLog.enabled true spark.history.fs.logDirectory file:///c:/logs/dir/path

• After configuration is finished run the following command from %SPARK_HOME%

  bin\spark-class.cmd org.apache.spark.deploy.history.HistoryServer

• The output should be something like that:

  16/07/22 18:51:23 INFO Utils: Successfully started service on port 18080. 16/07/22 18:51:23 INFO HistoryServer: Started HistoryServer at http://10.0.240.108:18080 16/07/22 18:52:09 INFO ShutdownHookManager: Shutdown hook called

Hope that it helps! :-)

Answer 2

in case any one gets the floowing exception:

17/05/12 20:27:50 ERROR FsHistoryProvider: Exception encountered when attempting
 to load application log file:/C:/Spark/Logs/spark--org.apache.spark.deploy.hist
ory.HistoryServer-1-Arsalan-PC.out
java.lang.IllegalArgumentException: Codec [out] is not available. Consider setti
ng spark.io.compression.codec=snappy
        at org.apache.spark.io.CompressionCodec$$anonfun$createCodec$1.apply(Com

Just go to SparkHome/config/spark-defaults.conf and set spark.eventLog.compress false