1. 技术文章
  2. how to use Kafka 0.8 Log4j appender

how to use Kafka 0.8 Log4j appender

由 a 夏天 提交于 2019-12-04 03:55:27

I think Jonas has identified the problem, that is the Kafka producer logging is also getting logged to the Kafka appender causing an infinite loop and eventual stack overflow (no pun intended) You can configure all Kafka logs to go to a different appender. The following shows sending the output to stdout:

log4j.logger.kafka=INFO, stdout

So you should end up with the following in your log4j.properties

log4j.rootLogger=INFO, stdout, KAFKA
log4j.logger.kafka=INFO, stdout
log4j.logger.HelloWorld=INFO, KAFKA

I have been able to generate events via log4j in Kafka 0.8.2.2. Here is my log4j configuration:

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">

<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">

   <appender name="console" class="org.apache.log4j.ConsoleAppender">
      <param name="Target" value="System.out" />
      <layout class="org.apache.log4j.PatternLayout">
         <param name="ConversionPattern" value="%-5p %c{1} - %m%n" />
      </layout>
   </appender>
   <appender name="fileAppender" class="org.apache.log4j.RollingFileAppender">
      <param name="Threshold" value="INFO" />
      <param name="MaxBackupIndex" value="100" />
      <param name="File" value="/tmp/agna-LogFile.log" />
      <layout class="org.apache.log4j.PatternLayout">
         <param name="ConversionPattern" value="%d  %-5p  [%c{1}] %m %n" />
      </layout>
   </appender>
   <appender name="kafkaAppender" class="kafka.producer.KafkaLog4jAppender">
      <param name="Topic" value="kafkatopic" />
      <param name="BrokerList" value="localhost:9092" />
      <param name="syncSend" value="true" />
      <layout class="org.apache.log4j.PatternLayout">
         <param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L %% - %m%n" />
      </layout>
   </appender>
   <logger name="org.apache.kafka">
      <level value="error" />
      <appender-ref ref="console" />
   </logger>
   <logger name="com.example.kafkaLogger">
      <level value="debug" />
      <appender-ref ref="kafkaAppender" />
   </logger>
   <root>
      <priority value="debug" />
      <appender-ref ref="console" />
   </root>
</log4j:configuration>

Here is the source code:

package com.example;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import org.json.simple.JSONArray;
import org.json.simple.JSONObject;
import java.util.Properties;
import java.util.concurrent.ExecutionException;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.KafkaProducer;

import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;

public class JsonProducer {
    static Logger defaultLogger = LoggerFactory.getLogger(JsonProducer.class);
    static Logger kafkaLogger = LoggerFactory.getLogger("com.example.kafkaLogger");

    public static void main(String args[]) {

        JsonProducer obj = new JsonProducer();

        String str = obj.getJsonObjAsString();

        // Use the logger
        kafkaLogger.info(str);

        try {
            // Construct and send message
            obj.constructAndSendMessage();
        } catch (InterruptedException e) {
            defaultLogger.error("Caught interrupted exception " + e);
        } catch (ExecutionException e) {
            defaultLogger.error("Caught execution exception " + e);
        }   
    }

    private String getJsonObjAsString() {
        JSONObject obj = new JSONObject();
        obj.put("name", "John");
        obj.put("age", new Integer(55));
        obj.put("address", "123 MainSt, Palatine, IL");

        JSONArray list = new JSONArray();
        list.add("msg 1");
        list.add("msg 2");
        list.add("msg 3");

        obj.put("messages", list);

        return obj.toJSONString();
    }

    private void constructAndSendMessage() throws InterruptedException, ExecutionException {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

        KafkaProducer<String, String> producer = new KafkaProducer<String, String>(props);

        boolean sync = false;
        String topic = "kafkatopic";
        String key = "mykey";
        String value = "myvalue1 mayvalue2 myvalue3";
        ProducerRecord<String, String> producerRecord = new ProducerRecord<String, String>(topic, key, value);
        if (sync) {
            producer.send(producerRecord).get();
        } else {
            producer.send(producerRecord);
        }
        producer.close();
    }
}

The whole project is a available under the following link:

https://github.com/ypant/kafka-json-producer.git

Try to set the appender async, like this: log4j.appender.KAFKA.ProducerType=async

Seems reasonable that it goes in to an infinite loop because the kafka producer has logging in itself..

标签
log4j
appender
apache-kafka
易学教程内所有资源均来自网络或用户发布的内容，如有违反法律规定的内容欢迎反馈！
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!