apache-kafka

问题 I am using Kafka 1.0.1 in my application and I have started using the Idempotent Producer feature that was introduced in 0.11, and I've having trouble understanding the ordering guarantees when using the Idempontent feature. My producer's configuration is: enable.idempotence = true max.in.flight.requests.per.connection = 5 retries = 50 acks = all According to the documentation: retries Setting a value greater than zero will cause the client to resend any record whose send fails with a

问题 I have a single source CSV file containing records of different sizes that pushes every record into one source topic. I want to split the records into different KStreams/KTables from that source topic. I have a pipeline for one table load, where I am pushing the record from the source topic into stream1 in delimited format and then pushing the records into another stream in AVRO format which is then pushed into JDBC sink connector that pushes the record into MySQL database. The pipeline needs

问题 I have a single source CSV file containing records of different sizes that pushes every record into one source topic. I want to split the records into different KStreams/KTables from that source topic. I have a pipeline for one table load, where I am pushing the record from the source topic into stream1 in delimited format and then pushing the records into another stream in AVRO format which is then pushed into JDBC sink connector that pushes the record into MySQL database. The pipeline needs

问题 i currently profiled my kafka producer spring boot application and found many "kafka-producer-network-thread"s running (47 in total). Which would never stop running, even when no data is sending. My application looks a bit like this: var kafkaSender = KafkaSender(kafkaTemplate, applicationProperties) kafkaSender.sendToKafka(json, rs.getString("KEY")) with the KafkaSender: @Service class KafkaSender(val kafkaTemplate: KafkaTemplate<String, String>, val applicationProperties:

问题 i currently profiled my kafka producer spring boot application and found many "kafka-producer-network-thread"s running (47 in total). Which would never stop running, even when no data is sending. My application looks a bit like this: var kafkaSender = KafkaSender(kafkaTemplate, applicationProperties) kafkaSender.sendToKafka(json, rs.getString("KEY")) with the KafkaSender: @Service class KafkaSender(val kafkaTemplate: KafkaTemplate<String, String>, val applicationProperties:

问题 How can I make sure if a topic is authorized or not ? I need this because, in my consumer I get the meta data for all the known topics and then do assign call. The metadata call doesn't give the un-authorized topics and non-existent topic. If a topic doesn't exist, I'll create one and if a topic is unauthorized, I have to fail. But I don't have a way to differentiate between non-existent and unauthorized topic. 回答1: You can try listing all the topics, if the topic exists it will be there in

问题 Closed . This question needs details or clarity. It is not currently accepting answers. Want to improve this question? Add details and clarify the problem by editing this post. Closed 5 months ago . Improve this question I am looking to deploy a spring boot app in an environment that only allows Kafka to be run by the aforementioned app. My app will be a Kafka producer and consumer. Is there a way to run an in memory instance on startup that can be used for purposes other than testing?

问题 Closed . This question needs details or clarity. It is not currently accepting answers. Want to improve this question? Add details and clarify the problem by editing this post. Closed 5 months ago . Improve this question I am looking to deploy a spring boot app in an environment that only allows Kafka to be run by the aforementioned app. My app will be a Kafka producer and consumer. Is there a way to run an in memory instance on startup that can be used for purposes other than testing?

问题 Closed . This question needs details or clarity. It is not currently accepting answers. Want to improve this question? Add details and clarify the problem by editing this post. Closed 5 months ago . Improve this question I am looking to deploy a spring boot app in an environment that only allows Kafka to be run by the aforementioned app. My app will be a Kafka producer and consumer. Is there a way to run an in memory instance on startup that can be used for purposes other than testing?

问题 Using the below configuration to simulate consumer going down /session timeout. How can we capture messages logged to the console by client - SESSTMOUT|rdkafka#consumer-1| [thrd:main] consumed message None: msg1: 0: first_topic: 0: None consumed message None: msg2: 1: first_topic: 0: None no message received by consumer no message received by consumer %4|1603348021.170|SESSTMOUT|rdkafka#consumer-1| [thrd:main]: Consumer group session timed out (in join-state started) after 10005 ms without a