Kafka - How to capture messages generated by kafka client background thread

两盒软妹~` 提交于 2021-02-05 09:26:10

问题


Using the below configuration to simulate consumer going down /session timeout. How can we capture messages logged to the console by client - SESSTMOUT|rdkafka#consumer-1| [thrd:main]

consumed message None: msg1: 0: first_topic: 0: None
consumed message None: msg2: 1: first_topic: 0: None
no message received by consumer
no message received by consumer
%4|1603348021.170|SESSTMOUT|rdkafka#consumer-1| [thrd:main]: Consumer group session timed out (in join-state started) after 10005 ms without a successful response from the group coordinator (broker 0, last error was Success): revoking assignment and rejoining group
no message received by consumer
no message received by consumer
no message received by consumer
%4|1603276138.721|MAXPOLL|rdkafka#consumer-1| [thrd:main]: Application maximum poll interval (30000ms) exceeded by 7ms (adjust max.poll.interval.ms for long-running message processing): leaving group
error from consumer KafkaError{code=_MAX_POLL_EXCEEDED,val=-147,str="Application maximum poll interval (30000ms) exceeded by 7ms"}
from confluent_kafka import Consumer
def consume():
    c = Consumer({"bootstrap.servers": "localhost:9092", 
                  "group.id": "group1",
                  "enable.auto.commit": False,
                  "auto.offset.reset": "earliest",
                  "max.poll.interval.ms": 30000,
                  "session.timeout.ms": 10000,
                  "heartbeat.interval.ms": 15000
                  })
    c.subscribe(["first_topic"])
    while True:
        message = c.poll(1.0)
        if message is None:
            print("no message received by consumer")
        elif message.error() is not None:
            print(f"error from consumer {message.error()}")
        else:
            print(f"consumed message {message.key()}: {message.value().decode('utf-8')}: {message.offset()}: {message.topic()}: {message.partition()}: {message.headers()}")
        time.sleep(10)

来源:https://stackoverflow.com/questions/64479665/kafka-how-to-capture-messages-generated-by-kafka-client-background-thread

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!