How to ensure a kafka cluster is fully up?

Question

We have five node cluster running in production with 3 zookeepers - all are VMs. We have to restart the cluster often for some hardware patching.

We have written an ans

Answer 1

From the Java API, you can use AdminClient#describeCluster, which can return you the list of nodes in the cluster currently known to the Controller.

---

Also, do not do broker rolling restarts like that. You should first identify which is the Contoller, then shut that down last.

Answer 2

Once all brokers have been started we can use following cmds, to check whether they have formed a cluster or not.

• From kafka-1 run the following command against the rest of the brokers, i.e. i = 2, 3, 4 and 5:

  • nc -vz kafka-i 9092 [It should return connection succeeded]

• tail the server.log in each broker node. It should give the info about the cluster.

• From Kafka bin directory, You can periodically run ./zookeeper-shell.sh zk_host:zk_port and execute ls /brokers/ids. It should gives you five entries, e.g. [0, 1, 2, 3, 4] if all 5 brokers have registered to the zookeeper.

One dirty (less involved) hack might be to create a test topic with 5 partitions, and wait until each broker gets 1 partition to itself.