Kafka Consumer does not receive data when one of the brokers is down

Question

Kafka Quickstart
Using Kafka v2.1.0 on RHEL v6.9

Consumer fails to receive data when one of the Kafka brokers is down.

Steps performed:
1. Start

Answer 1

In initial versions of Kafka, offset was being managed at zookeeper, but Kafka has continuously evolved over the time introducing lot of new features. Now Kafka manages the offset in a topic __consumer_offsets.

You can think of a scenario where you created a topic with a replication factor of 1. In case the broker goes down the data is only on that Kafka node which is down. So you can't get this data. Same analogy applies to __consumer_offsets topic.

You need to revisit the server.properties in order to get features you are expecting. But in case you still wanna consume the messages from the replica partition, you may need to re-start the console consumer with --from-beginning true