Kafka Connect and Kafka Broker version compatibility

心不动则不痛 提交于 2019-12-11 16:41:38

问题


We have a "Kerberized Kafka cluster" running brokers version Apache Kafka 0.11.This cluster is managed by a different team and we dont have any control over this.

We are now trying to install Kafka Connect cluster on our own K8S cluster.

We were following this compatibility matrix https://docs.confluent.io/current/installation/versions-interoperability.html

According to this, we had to stick with Confluent Platform 3.3.3 images for Schema Registry and Kafka Connect pods since the Brokers version is 0.11

Also we want to enable Connect JMX metrics which was implemented here : https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework

According to the documentation these JMX metrics are supported in Kafka 1.0.0

Questions are:

1) Can we use latest Confluent images (eg: version 5.2.0) with Kafka Brokers version 0.11?

2) With current setup (Confluent 3.3.3) I have enabled all the JMX metrics but we don't see any Kafka Connector based metrics. I see general Kafka Connect metrics, but not metrics per each Connector(eg. Source and Sink metrics).Is this because of the old CP version we use ?


回答1:


Kafka Connect Workers that are included in Confluent Platform 3.2 and later are compatible with any Kafka broker that is included in Confluent Platform 3.0 and later

Note: Kafka Connect is not a Confluent Platform feature and is built over plain Apache Kafka API. Therefore, Connect instances using Kafka 1.0 and 2.0 client libraries still work with Kafka 0.11 brokers for basic communication protocols

If you're having Kerberos issues, those are likely unrelated

I see general Kafka Connect metrics, but not metrics per each Connector

In general, there is no such per connector metric such as "number of records read/written to a database", or at least, it would largely depend if the connectors actual implement their own metrics, but yes, the 3.3.x releases are all based on Kafka 0.11, not 1.0, where more Connect API metrics were added



来源:https://stackoverflow.com/questions/56519301/kafka-connect-and-kafka-broker-version-compatibility

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!