Why does a Kafka consumer take a long time to start consuming?

Question

We start a Kafka consumer, listening on a topic which may not yet be created (topic auto creation is enabled though).

Not long thereafter a producer is publishing m

Answer 1

This is probably due to the default value of the parameter metadata.max.age.ms which controls how often the consumer forces a refresh of metadata for a topic.

What happens when you start the consumer up with a non existing topic is that the brokers autocreate this topic, but this takes a little bit of time with leader election etc., so when your consumer requests metadata for that topic it gets a LEADER_NOT_AVAILABLE warning and can't fetch any messages. After the timeout mentioned above is reached the consumer refreshes metadata, successfully this time around and starts reading messages. This is not dependent on a producer writing messages to the topic, it is purely a consumer thing.

If you start your consumer with for example 1000ms timeout, you should see a much shorter delay until messages are consumed.

Also, if you create topics up front, or start the producer before the consumer, this behavior should not happen at all.