Apache spark: setting spark.eventLog.enabled and spark.eventLog.dir at submit or Spark start

Question

I would like to set spark.eventLog.enabled and spark.eventLog.dir at the spark-submit or start-all level -- not require it to

Answer 1

I solved the problem, yet strangely I had tried this before... All the same, now it seems like a stable solution:

Create a directory in HDFS for logging, say /eventLogging

hdfs dfs -mkdir /eventLogging

Then spark-shell or spark-submit (or whatever) can be run with the following options:

--conf spark.eventLog.enabled=true --conf spark.eventLog.dir=hdfs://:8020/eventLogging

such as:

spark-shell --conf spark.eventLog.enabled=true --conf spark.eventLog.dir=hdfs://:8020/eventLogging