Getting Kafka usage details

问题

I am trying to find ways to get current usage statistics for my kafka cluster. I am looking to collect following information:

1. Number of topics in kafka cluster
2. Number of partitions per kafka broker
3. Number of active consumers and producers
4. Number of client connections per kafka broker
5. Number of messages on each partition, size of disk etc.
6. Lagging replicas, consumer lag etc.
7. Active consumer groups

Any other statistics that can and should be collected, currently I am looking at collecting the above stats.

I can get 1 and 2 using zookeeper utilities but I am lost on rest. I have looked at mbeans in Jconsole but didn't find anything about above. I also tried JmxTool to get these mbeans using regex based expression but that also didn't work.

I am using Kafka v2.1 and using new consumer api so zookeeper doesn't have any information about consumers.

Any pointers would be great help!

回答1:

Might as well use https://github.com/yahoo/kafka-manager or https://github.com/linkedin/cruise-control to get this information.

There are scripts under $KAFKA_HOME/bin which can help you.

Number of topics in kafka cluster

./kafka-topics.sh --zookeeper localhost:2181 --list

Number of partitions per kafka broker

./kafka-topics.sh --zookeeper localhost:2181 --describe

Number of messages on each partition, size of disk etc.

./kafka-log-dirs.sh --describe --bootstrap-server localhost:9092

Lagging replicas, consumer lag etc.

./kafka-consumer-group.sh --bootstrap-server localhost:9092 --group $GROUP_NAME --describe

Active consumer groups Number of active consumers and producers

You can't get active producer. Know existing producers for a kafka topic

./kafka-consumer-group.sh --bootstrap-server localhost:9092 --list

Number of client connections per kafka broker

./

来源：https://stackoverflow.com/questions/54652733/getting-kafka-usage-details