Python Kafka consumer missing to poll some messages

Question

The code for my Kafka consumer looks like this

def read_messages_from_kafka():
    topic = \'my-topic\'
    consumer = KafkaConsumer(
        bootstrap_servers=[         


        

Answer 1

There is a problem of missing messages in your Kafka client. I found solution here:

while True:
    raw_messages = consumer.poll(timeout_ms=1000, max_records=5000)
    for topic_partition, messages in raw_messages.items():
        application_message = json.loads(message.value.decode())

Also there is another Kafka client exists: confluent_kafka. It has no such problem.