Kafka Consumer outputs excessive DEBUG statements to console (ecilpse)

Question

I\'m running some sample code from http://www.javaworld.com/article/3060078/big-data/big-data-messaging-with-kafka-part-1.html?page=2, and the kafkaconsumer is consuming from to

Answer 1

create new config xml file

src/main/resources/logback.xml

<configuration>
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>
    <logger name="org.apache.kafka" level="WARN"/>
    <logger name="org.apache.kafka.common.metrics" level="WARN"/>
    <root level="warn">
        <appender-ref ref="STDOUT" />
    </root>
</configuration>

Answer 2

Not sure if you are talking about kafka-console-consumer commands, if yes this is what I did:

[training@confluent-training-vm ~]$ cd /etc/kafka
[training@confluent-training-vm kafka]$ grep DEBUG *.properties
log4j.properties:# Change to DEBUG or TRACE to enable request logging
log4j.properties:# Access denials are logged at INFO level, change to DEBUG to also 
log allowed accesses
tools-log4j.properties:log4j.rootLogger=DEBUG, stderr

So, you just need to edit /etc/kafka/tools-log4j.properties file and remove DEBUG (or replace it for exmaple by INFO and WARM log levels on above line

tools-log4j.properties:log4j.rootLogger=INFO, WARM, stderr

Answer 3

Just modify the logging level of the chatty class (chatty interaction). Since in your logs you see log entries originating from org.apache.kafka.clients.consumer.internals.Fetcher you can simply adjust the logging level for that logger by adding following line to log4j.properties:

log4j.logger.org.apache.kafka.clients.consumer.internals.Fetcher=WARN

... or any wider catching logger since these are name spaced:

# adjusting logging for entire Kafka
log4j.logger.org.apache.kafka=WARN

Hope this helps

Answer 4

I found the solution under another question which deals with Kafka producers, but it's essentially the same problem: https://stackoverflow.com/a/49532152/2380553

So I just run the following 3 lines at the beginning of my program:

org.apache.log4j.Logger.getLogger("org").setLevel(Level.WARN);
org.apache.log4j.Logger.getLogger("akka").setLevel(Level.WARN);
org.apache.log4j.Logger.getLogger("kafka").setLevel(Level.WARN);