SparkContext Error - File not found /tmp/spark-events does not exist

Question

Running a Python Spark Application via API call - On submitting the Application - response - Failed SSH into the Worker

My python application exists in

Answer 1

Use spark.eventLog.dir for client/driver program

spark.eventLog.dir=/usr/local/spark/history

and use spark.history.fs.logDirectory for history server

spark.history.fs.logDirectory=/usr/local/spark/history

as mentioned in: How to enable spark-history server for standalone cluster non hdfs mode

At least as per Spark version 2.2.1