Log4j RollingFileAppender not adding mapper and reducer logs to file

问题

We would like our application logs to be printed to files on the local nodes. We're using Log4j's RollingFileAppender.

Our log4j.properties file is as follows:

ODS.LOG.DIR=/var/log/appLogs
ODS.LOG.INFO.FILE=application.log
ODS.LOG.ERROR.FILE=application_error.log
# Root logger option
log4j.rootLogger=ERROR, console
log4j.logger.com.ournamespace=ERROR, APP_APPENDER, ERROR_APPENDER

#
# console
# Add "console" to rootlogger above if you want to use this
#

log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.out
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss}-%r %p %c{5}: %m%n

# Direct log messages to a log file
log4j.appender.APP_APPENDER=org.apache.log4j.RollingFileAppender
log4j.appender.APP_APPENDER.File=${ODS.LOG.DIR}/${ODS.LOG.INFO.FILE}
log4j.appender.APP_APPENDER.MaxFileSize=200MB
log4j.appender.APP_APPENDER.MaxBackupIndex=30
log4j.appender.APP_APPENDER.layout=org.apache.log4j.PatternLayout
log4j.appender.APP_APPENDER.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss}-%r %p %c{10}: %m%n


log4j.appender.ERROR_APPENDER=org.apache.log4j.RollingFileAppender
log4j.appender.ERROR_APPENDER.Threshold=ERROR
log4j.appender.ERROR_APPENDER.File=${ODS.LOG.DIR}/${ODS.LOG.ERROR.FILE}
log4j.appender.ERROR_APPENDER.MaxFileSize=200MB
log4j.appender.ERROR_APPENDER.MaxBackupIndex=90
log4j.appender.ERROR_APPENDER.layout=org.apache.log4j.PatternLayout
log4j.appender.ERROR_APPENDER.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss}-%r %p %c{10}: %m%n

Our MR Driver logs are appearing in /var/log/appLogs directory but our mapper and reducer logs dont show up in this directory.

We have copied our log4j.properties snippet to hdfs-log4j, yarn-log4j, hbase-log4j and zookeeper-log4j using Ambari (Hortonworks Data Platform). Our MR jobs typically use HBase input and output format classes.

What changes do we need to make to have the mapper and reducer logs appear in the /var/log/appLogs directory as well?

Edit: The logs are appearing in the JobHistory UI syslog but they aren't being added to the log file. What are we missing?

回答1:

For example to configure log4j you can call PropertyConfigurator.configure(properties); from your code e.g. in mapper/reducer setup method.

This is example with properties stored on hdfs:

        InputStream is = fs.open(log4jPropertiesPath);
        Properties properties = new Properties();
        properties.load(is);
        PropertyConfigurator.configure(properties);

where fs is FileSystem object and log4jPropertiesPath is path on hdfs.

来源：https://stackoverflow.com/questions/49797255/log4j-rollingfileappender-not-adding-mapper-and-reducer-logs-to-file

标签

Hadoop

logging

MapReduce

log4j

hortonworks-data-platform