Spark executor logs on YARN

Question

I\'m launching a distributed Spark application in YARN client mode, on a Cloudera cluster. After some time I see some errors on Cloudera Manager. Some executors get disconne

Answer 1

Check NodeManager's yarn.nodemanager.log-dir property. It's the log location of when Spark executor container is running.

Note that when the application finishes NodeManager may remove the files (Log Aggregation). Check this document for detail. http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/