问题
Need to check for tool to monitor Kafka in production. Also tool does not need to have license or heavy hardware. In particular I need a tool to evaluate consumer offset on topic,health of topic.
回答1:
Landoop
Landoop enhances Kafka with User Interface, streaming SQL engine and cluster monitoring. It enables faster monitoring of Kafka data pipelines.
They provide a free all-in-one docker (LENSES Box) which can serve a single broker for up to 25M messages. Note that this is recommended for development environments.
Confluent
Another option is Confluent Enterprise which is a Kafka distribution for production environments. It also includes Control Centre, which is a management system for Apache-Kafka that enables cluster monitoring and management from a User Interface.
Yahoo Kafka Manager
Kafka's Manager is a tool for monitoring Kafka offering less functionality compared to the aforementioned tools.
KafDrop
KafDrop is a UI for monitoring Apache Kafka clusters. The tool displays information such as brokers, topics, partitions, and even lets you view messages. It is a light weight application that runs on Spring Boot and requires very little configuration.
LinkedIn Burrow
Burrow is a monitoring companion for Apache Kafka that provides consumer lag checking as a service without the need for specifying thresholds. It monitors committed offsets for all consumers and calculates the status of those consumers on demand. An HTTP endpoint is provided to request status on demand, as well as provide other Kafka cluster information. There are also configurable notifiers that can send status out via email or HTTP calls to another service.
Kafka's Tool
Kafka's Tool is a GUI application for managing and using Apache-Kafka clusters. It provides an intuitive UI that allows one to quickly view objects within a Kafka cluster as well as the messages stored in the topics of the cluster. It contains features geared towards both developers and administrators.
I am afraid that you will not find any free products suitable for production environments but if you cannot afford it then go for Yahoo Kafka Manager, LinkedIn Burrow or KafDrop. Confluent's and Landoop's products are the best out there but unfortunately, they require licensing.
For more details, you can refer to my blog post Overview of UI Monitoring tools for Apache Kafka Clusters.
回答2:
There are several free monitoring solutions, but all have their limitations. For example, Burrow, Kafka Manager, Kafdrop, and you can brew your own with the JMX metrics plus something like InfluxDB and Grafana.
Confluent Control Center is perfectly suited to your requirement, however it does require a licence. There is a 30 day free trial available. It takes things beyond just monitoring a bunch of metrics, to actually monitoring individual topics and the messages flowing through and checksumming each to ensure accurate reporting of consumption.
回答3:
Kafka is producing quite much information in/via JMX. You can use any JMX client to get it. Even internally in your app you can forward Kafka JMX information to your JMX MBeans to build up, so to say, JMX Facade.
Some concrete proposals:
- Grafana + Prometheus works great with Kafka. You can also dowload from the internet some ready to use dashboards (free);
- I have been using also hawtio to see JMX information and it is doing the job as well (free);
- Confluent offers also monitoring;
- Also there is some speciall support in Dynatrace.
回答4:
I’ve just release an opensource GUI for kafka : https://github.com/tchiotludo/kafkahq Any feedback are appreciate !
preview
You can also find some other UI here : https://github.com/monksy/awesome-kafka/blob/master/tools.md#dashboards
回答5:
It depends on the extent of monitoring and level of automation. Kafka Manager is good place to start for insights/observability into the health of the cluster, throughput, latency, etc. and some basic admin operations, but pretty soon you'll be reaching for more. For instance, do you need to monitor the health of your consumers? Do you need to view the contents of topics to debug applications, perform post-mortem, trace message flow, etc.? How will you handle bad/malformed messages, 'poisoned' topics, etc.? I'd suggest adding a few more items into your toolbox:
- Burrow - monitors consumer group lag and raises alerts if the consumers fall behind.
- Kafdrop 3 - provides broker information and lets you browse topic contents (both text and Avro formats). This is a more recent fork of the original Kafdrop project that supports latest versions of Kafka. DISCLAIMER: I am one of the maintainers the project.
- Prometheus and Grafana - there's some overlap with Manager/Monitor, but Prometheus/Grafana are much more refined, have substantial community support and are easier to integrate with other tools, automatically raise alerts, etc.
All of the above are open source and don't require licensing. I didn't include Confluent's, Landoop's or KaDeck tools into the list simply because they are commercial and require licensing. And they don't really give you much over the OSS alternatives, IMHO.
Having run Kafka now for some time as a main messaging/event backbone for large microservices deployments since 2014, can't stress enough the importance of going beyond just the basic broker-level checks and looking at the entire end-to-end message path, from the producer down to the consumers. Ironically, we found most of the time when we felt the brokers were playing up, it was actually the applications that were badly written. We tend to use a combination of tools, down to writing our own basic 'canary' applications that will pretend to act as message producers and consumers and give us a leading indication when anything is untoward.
Kafka does ship with some basic CLI tools of its own in $KAFKA_HOME/bin, such as kafka-console-consumer.sh (for reading the topic) and other tools such as kafka-topics.sh and kafka-configs.sh for administrative tasks. There is also a separate tool — Kafkacat — for browsing topics. But in my experience these get used as a last resort, for example if you lose connectivity to the cluster and need to shell onto one of the broker nodes, then these do come in very useful. But they aren't monitoring tools per se.
来源：https://stackoverflow.com/questions/49043596/kafka-monitoring-tool-in-production