Not able to run Kafka Connect in distributed mode - Error while attempting to create/ find topics 'connect-offsets'

微笑、不失礼 提交于 2021-02-19 06:23:30

问题


[2017-08-31 10:15:20,715] WARN The configuration 'internal.key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:231) [2017-08-31 10:15:20,715] WARN The configuration 'status.storage.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:231) [2017-08-31 10:15:20,715] WARN The configuration 'internal.value.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:231) [2017-08-31 10:15:20,715] WARN The configuration 'value.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:231) [2017-08-31 10:15:20,715] WARN The configuration 'internal.value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:231) [2017-08-31 10:15:20,715] WARN The configuration 'offset.storage.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:231) [2017-08-31 10:15:20,716] WARN The configuration 'offset.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:231) [2017-08-31 10:15:20,716] WARN The configuration 'value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:231) [2017-08-31 10:15:20,716] WARN The configuration 'key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:231) [2017-08-31 10:15:20,768] INFO jetty-9.2.15.v20160210 (org.eclipse.jetty.server.Server:327) Aug 31, 2017 10:15:21 AM org.glassfish.jersey.internal.Errors logErrors WARNING: The following warnings have been detected: WARNING: The (sub)resource method listConnectors in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation. WARNING: The (sub)resource method createConnector in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation. WARNING: The (sub)resource method listConnectorPlugins in org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource contains empty path annotation. WARNING: The (sub)resource method serverInfo in org.apache.kafka.connect.runtime.rest.resources.RootResource contains empty path annotation.

[2017-08-31 10:15:21,142] INFO Started o.e.j.s.ServletContextHandler@63f02de0{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:744) [2017-08-31 10:15:21,149] INFO Started ServerConnector@61b06738{HTTP/1.1}{0.0.0.0:8083} (org.eclipse.jetty.server.ServerConnector:266) [2017-08-31 10:15:21,149] INFO Started @13584ms (org.eclipse.jetty.server.Server:379) [2017-08-31 10:15:21,150] INFO REST server listening at http://172.28.193.126:8083/, advertising URL http://172.28.193.126:8083/ (org.apache.kafka.connect.runtime.rest.RestServer:150) [2017-08-31 10:15:21,150] INFO Kafka Connect started (org.apache.kafka.connect.runtime.Connect:55) [2017-08-31 10:15:21,468] ERROR Uncaught exception in herder work thread, exiting: (org.apache.kafka.connect.runtime.distributed.DistributedHerder:206) org.apache.kafka.connect.errors.ConnectException: Error while attempting to create/find topic(s) 'connect-offsets' at org.apache.kafka.connect.util.TopicAdmin.createTopics(TopicAdmin.java:245) at org.apache.kafka.connect.storage.KafkaOffsetBackingStore$1.run(KafkaOffsetBackingStore.java:99) at org.apache.kafka.connect.util.KafkaBasedLog.start(KafkaBasedLog.java:126) at org.apache.kafka.connect.storage.KafkaOffsetBackingStore.start(KafkaOffsetBackingStore.java:109) at org.apache.kafka.connect.runtime.Worker.start(Worker.java:146) at org.apache.kafka.connect.runtime.AbstractHerder.startServices(AbstractHerder.java:99) at org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:194) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89) at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:213) at org.apache.kafka.connect.util.TopicAdmin.createTopics(TopicAdmin.java:226)


回答1:


Kafka Connect 0.11.0.0 cannot create the topics on brokers older than 0.10.1.0, since the protocol used in those older brokers do not support topic management.

So, if you have an older broker, you should either manually create the topics or rely upon broker auto-creation. Unfortunately, a bug in Kafka Connect 0.11.0.0 prevents auto-creation, and so until 0.11.0.1 is available you will have to manually create the topics.




回答2:


When starting up Kafka Connect in distributed mode, it will try to create or expect several topics to be present in Kafka. In my case auto topic creation had been turned off so Kafka Connect could not create topics in the environment I was running it in. Creating these topics myself solved the issue. You would have specified the names of these topics in your properties file (below).

offset.storage.topic

config.storage.topic

status.storage.topic



来源:https://stackoverflow.com/questions/45987021/not-able-to-run-kafka-connect-in-distributed-mode-error-while-attempting-to-cr

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!