spark-submit, how to specify log4j.properties

Question

In spark-submit, how to specify log4j.properties ?

Here is my script. I have tried all of combinations and even just use one local node. but looks like the log4j.propert

Answer 1

How to pass local log4j.properties file

As I see from your script you want to:

1. Pass local log4j.properties to executors
2. Use this file for node's configuration.

---

Note two things about --files settings:

1. Files uploaded to spark-cluster with --files will be available at root dir, so there is no need to add any path in file:log4j.properties.
2. Files listed in --files must be provided with absolute path!

Fixing your snippet is very easy now:

current_dir=/tmp
log4j_setting="-Dlog4j.configuration=file:log4j.properties"

spark-submit \
...
--conf "spark.driver.extraJavaOptions=${log4j_setting}" \
--conf "spark.executor.extraJavaOptions=${log4j_setting}" \
--class "my.AppMain" \
--files ${current_dir}/log4j.properties \
...
$current_dir/my-app-SNAPSHOT-assembly.jar

---

Need more?

If you would like to read about other ways of configuring logging while using spark-submit, please visit my other detailed answer: https://stackoverflow.com/a/55596389/1549135