Consumer not receiving messages, kafka console, new consumer api, Kafka 0.9

Question

I am doing the Kafka Quickstart for Kafka 0.9.0.0.

I have zookeeper listening at localhost:2181 because I ran

bin/zookeeper-server-start.sh          


        

Answer 1

For me the solution described in this thread worked - https://stackoverflow.com/a/51540528/7568227

Check if

offsets.topic.replication.factor

(or probably other config parameters related to replication) is not higher than the number of brokers. That was the problem in my case.

There was no need to use --partition 0 anymore after this fix.

Otherwise I recommend to follow the debugging procedure described in the mentioned thread.

Answer 2

replication factor must be at least 3

./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic test 

Answer 3

I got the same problem, now I have figured out.

When you use --zookeeper, it is supposed to be provided with zookeeper address as parameter.

When you use --bootstrap-server, it is supposed to be provided with broker address as parameter.

Answer 4

In my case, this doesn't work

kafka-console-consumer --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic

and this works

kafka-console-consumer --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic --partition 0

because the topic __consumer_offsets was located on the unaccessible broker. Basically, I'd forgotten to replicate it. Relocating __consumer_offsets solved my issue.

Answer 5

In my case it didn't worked using either approaches then I also increased the log level to DEBUG at config/log4j.properties, started the console consumer

./bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --from-beginning --topic MY_TOPIC

Then got the log below

[2018-03-11 12:11:25,711] DEBUG [MetadataCache brokerId=10] Error while fetching metadata for MY_TOPIC-3: leader not available (kafka.server.MetadataCache)

The point here is that I have two kafka nodes but one is down, by some reason by default kafka-console consumer will not consume if there is some partition not available because the node is down (the partition 3 in that case). It doesn't happen in my application.

Possible solutions are

• Startup the down brokers
• Delete the topic and create it again that way all partitions will be placed at the online broker node

Answer 6

Your localhost is the foo here. if you replace the localhost word for the actual hostname, it should work.

like this:

producer

./bin/kafka-console-producer.sh --broker-list \
sandbox-hdp.hortonworks.com:9092 --topic test

consumer:

./bin/kafka-console-consumer.sh --topic test --from-beginning \    
--bootstrap-server bin/kafka-console-consumer.sh --new-consumer \
--topic test --from-beginning \
--bootstrap-server localhost:9092