How to set up logging level for Spark application in IntelliJ IDEA?

Question

I\'m working on a Scala project in IntelliJ that was created through SBT. The project has Spark as one of its dependencies. I\'m still in the development phase so everything is

Answer 1

Spark by default logs almost everything you would like to see in the logs, however, if you need to change the logging behaviour, you can edit log4j.properties in the conf directory of your Apache Spark configuration. If you're using a prebuilt version, you can find it in /home/coco/Applications/spark-1.4.0-bin-hadoop2.6/conf directory. There is a template file "log4j.properties.template" that you have to copy to "log4j.properties" and edit it depending on your needs. I hope it helps.