Kafka consumer, very long rebalances

Question

We are running a 3 broker Kafka 0.10.0.1 cluster. We have a java app which spawns many consumer threads consuming from different topics. For every topic we have specified differ

Answer 1

Your consumer configuration seems reasonable. I would advise trying three things:

• Try to spawn a single consumer thread, and assign it only one of the topics you're trying to consume from. That single thread should get all partitions for that topic assigned, and it should immediately start receiving all the data. You can try to print out the partition and message offset, as well as content, to validate that it's getting all the data.
• Once you validate that's working, spawn a single consumer thread, and assign it all the topics you're trying to consume from. Do the same validation printing out the messages.
• Finally, if that is working fine, start adding consumer threads one by one, and see if you start getting pauses when consuming.

That should allow you to pinpoint where the problem is. If you're able to consume everything with a single thread, but not with multiple threads, then your threading mechanism/pooling might have issues.