Is it possible to get the current spark context settings in PySpark?

Question

I\'m trying to get the path to spark.worker.dir for the current sparkcontext.

If I explicitly set it as a config param, I can read

Answer 1

You can use:

sc.sparkContext.getConf.getAll

For example, I often have the following at the top of my Spark programs:

logger.info(sc.sparkContext.getConf.getAll.mkString("\n"))