apache-kafka

问题 We are currently trying to use Debezium for capturing changes of 4 tables in a Postgres database. We are currently aware that for this use case we could use a kafka-streams app to join/aggregate tables for a KTable, however we want to keep kafka-stream topology simple, so the idea would be to use a Materialized View from Postgres and capture it changes. Is it possible to do this, if so than how should we configure it? Kafka Connect Source Configuration: { "name": "campaign-db-source-connector

问题 I have following function, that does not compile: private def save(pea: KStream[String, String]): Unit = { pea .groupByKey() .aggregate(() => """{folder: ""}""", (_: String, _: String, value: String) => value, EventStoreTopology.Store) } the error message is: [error] [VR](x$1: org.apache.kafka.streams.kstream.Initializer[VR], x$2: org.apache.kafka.streams.kstream.Aggregator[_ >: String, _ >: String, VR], x$3: org.apache.kafka.streams.processor.StateStoreSupplier[org.apache.kafka.streams.state

问题 I have a Spark streaming application reading from Kafka. I am running it from EMR. Recently I implemented Kafka SSL. I am creating the Kafka client as shown below. I am getting a strange error running the application when it tries to read the truststore file. Error is: - Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: java.io.FileNotFoundException: /tmp/kafka.client.truststore.jks (No such file or directory) What is causing this issue?

问题 I have a Spark streaming application reading from Kafka. I am running it from EMR. Recently I implemented Kafka SSL. I am creating the Kafka client as shown below. I am getting a strange error running the application when it tries to read the truststore file. Error is: - Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: java.io.FileNotFoundException: /tmp/kafka.client.truststore.jks (No such file or directory) What is causing this issue?

问题 I came across the following article on how to use the schema registry available in the confluent platform. https://docs.confluent.io/platform/current/schema-registry/schema-validation.html According to that article, we can specify confluent.schema.registry.url in server.properties to point Kafka to the schema registry. My question is, is it possible to point a Kafka cluster which is not a part of confluent platform deployment, to a schema registry using confluent.schema.registry.url ? 回答1:

问题 I am trying to schedule my consumption process from a single partition topic. I can start it using endpointlistenerregistry.start() but I want to stop it after I have consumed all the messages in current partition i.e. when I reach to last offset in current partition. Production into the topic is done after I have finished the consumption and close it. How should I achieve the assurance that I have read all the messages till the time I started scheduler and stop my consumer ? I am using

问题 I am trying to schedule my consumption process from a single partition topic. I can start it using endpointlistenerregistry.start() but I want to stop it after I have consumed all the messages in current partition i.e. when I reach to last offset in current partition. Production into the topic is done after I have finished the consumption and close it. How should I achieve the assurance that I have read all the messages till the time I started scheduler and stop my consumer ? I am using

问题 I want to read all the messages starting from a specific time in kafka. Say I want to read all messages between 0600 to 0800 Request messages between two timestamps from Kafka suggests the solution as the usage of offsetsForTimes. Problem with that solution is : If say my consumer is switched on everyday at 1300. The consumer would not have read any messages that day, which effectively means no offset was committed at/after 0600, which means offsetsForTimes(< partitionname > , <0600 for that

问题 I have a Flink session cluster (Job Manager + Task Manager), version 1.11.1, with configured log4j-console.properties to include Kafka appender. In addition, in both Job Manager and Task Manager I'm enabling flink-s3-fs-hadoop built-in plugin. I've added kafka-clients jar to the flink/lib directory, which is necessary for the container to be running. But I'm still getting the below class loading error when the S3 plugin is being instantiated (and initializing the logger). Caused by: org

问题 I have a Quarkus project with Smallrye reactive messaging based on Kafka. Since I want to work with a "complex pojo" I need a custom de-/serializer. I'd like to make those two classes CDI beans so I can inject and use my custom logger, which is a CDI bean. Is there a way to achieve this? Right now my injected logger object is simply null: import org.apache.kafka.common.serialization.Serializer; import javax.enterprise.context.ApplicationScoped; import javax.inject.Inject; @ApplicationScoped