spark-submit, how to specify log4j.properties

Question

In spark-submit, how to specify log4j.properties ?

Here is my script. I have tried all of combinations and even just use one local node. but looks like the log4j.propert

Answer 1

Pay attention the Spark worker is not your Java application, so you can't use a log4j.properties file from the class-path.

To understand how Spark on YARN will read a log4j.properties file, you can use the log4j.debug=true flag:

spark.executor.extraJavaOptions=-Dlog4j.debug=true

Most of the time, the error is that the file is not found/available from the worker YARN container. There is a very useful Spark directive that allows to share file: --files.

--files "./log4j.properties"

This will make this file available from all your driver/workers. Add Java extra options:

-Dlog4j.configuration=log4j.properties

Et voilà!

log4j: Using URL [file:/var/log/ambari-server/hadoop/yarn/local/usercache/hdfs/appcache/application_1524817715596_3370/container_e52_1524817715596_3370_01_000002/log4j.properties] for automatic log4j configuration.