Kafka consumer, very long rebalances

Question

We are running a 3 broker Kafka 0.10.0.1 cluster. We have a java app which spawns many consumer threads consuming from different topics. For every topic we have specified differ

Answer 1

I suspect your cluster version is at least 0.10.1.0 as I see max.poll.interval.ms in your consumer configuration which was introduced in this version.

Kafka 0.10.1.0 integrates KIP-62 which introduces a rebalance timeout set to max.poll.interval.ms and its default value is 5 minutes.

I guess if you don't want to wait timeout expiration during a rebalance, your consumers need to cleanly leave consumer group by calling close() method.