Why __consumer_offsets topic in kafka is not spreading to all the brokers?

Question

I have a 3 zk nodes cluster. And 7 kafka broker nodes.

So when I create any topic then I can set replica factor and no of partitions us

Answer 1

It sounds really strange to me. I deployed a 3 Kafka brokers cluster and my __consumer_offsets topic has 50 partitions (the default) split across the 3 brokers and having 3 (the default) as replicas. What you are describing could happen when you start with a single Kafka broker, create consumers for reading topics so that the __consumer_offsets is automatically created and it will be just on the broker 0 (the only one). After that, you add new Kafka brokers: without doing anything the __consumer_offsets will still stay on broker 0; you need to use Kafka a manual partition reassignment for doing that as mentioned above.

Answer 2

You will need to use Kafka partition reassignment. This is unfortunately a semi manual process currently: https://kafka.apache.org/documentation/#basic_ops_cluster_expansion

Answer 3

You need to set the "offsets.topic.replication.factor" to the desired number of replicas in the broker config.

offsets.topic.replication.factor - "The replication factor for the offsets topic (set higher to ensure availability). Internal topic creation will fail until the cluster size meets this replication factor requirement."

Reference - http://kafka.apache.org/documentation/#brokerconfigs