spark-submit, how to specify log4j.properties

Question

In spark-submit, how to specify log4j.properties ?

Here is my script. I have tried all of combinations and even just use one local node. but looks like the log4j.propert

Answer 1

1. Copy the spark-defaults.conf to a new app-spark-defaults.conf

2. Add -Dlog4j.configuration=file://log4j.properties to the spark.driver.extraJavaOptions in the app-spark-defaults.conf. For example:

   spark.driver.extraJavaOptions -XXOther_flag -Dlog4j.configuration=file://log4j.properties

3. Run your spark using --properties-file to the new conf file. For example :
   spark-submit --properties-file app-spark-defaults.conf --class my.app.class --master yarn --deploy-mode client ~/my-jar.jar