apache-kafka

问题 I currently maintain a financial application. While there are many calculations done in this financial application, one of the calculations is to determine 1) How much percentage of the total transaction amount does a new incoming transaction account for? 2) How much percentage of the total transaction amount for the given customer does the new transaction account for with respect to the same customer? For the sake of simplicity, let's assume that the transcation data will be cut off at 6 am

问题 ItemReader is reading data from DB2 and gave java object ClaimDto . Now the ClaimProcessor takes in the object of ClaimDto and return CompositeClaimRecord object which comprises of claimRecord1 and claimRecord2 which to be sent to two different Kafka topics. How to write claimRecord1 and claimRecord2 to topic1 and topic2 respectively. 回答1: Just write a custom ItemWriter that does exactly that. public class YourItemWriter implements ItemWriter<CompositeClaimRecord>` { private final ItemWriter

问题 ItemReader is reading data from DB2 and gave java object ClaimDto . Now the ClaimProcessor takes in the object of ClaimDto and return CompositeClaimRecord object which comprises of claimRecord1 and claimRecord2 which to be sent to two different Kafka topics. How to write claimRecord1 and claimRecord2 to topic1 and topic2 respectively. 回答1: Just write a custom ItemWriter that does exactly that. public class YourItemWriter implements ItemWriter<CompositeClaimRecord>` { private final ItemWriter

问题 I am trying to publish some message using kafka consumer with "replyingKafkaTemplate". My main prupose is to subscribe the message, modify it and sending back the modified message. I have tried increasing replyTimeout of replykafkaTemplate. But even that I am getting no response from subscriber. Producer console is showing following. I have tried increasing transaction timeout, request timeout. But nothing works for me. Any help would be appreciated. Thanks in advance These are my config

问题 Reading:Kafka Connect FileStreamSource ignores appended lines An Answer from 2018 states : Kafka Connect does not "watch" or "tail" a file. I don't believe it is documented anywhere that it does do that. It seems Kafka does now support this as reading: https://docs.confluent.io/5.5.0/connect/managing/configuring.html#standalone-example does state the file is watched: FileSource Connector The FileSource Connector reads data from a file and sends it to Apache Kafka®. Beyond the configurations

问题 I started last week a formation in Kafka and Storm at OpenClassRooms. During practical work, I encounter an error when I try to execute a JAR containing my java code for Storm. No problem when compiling the project in Java, no problem when packaging with maven, the problem only occurs when running the JAR theirman@vm-debian:/data/eclipse-workspace/velos$ storm jar target/velos-1.0-SNAPSHOT.jar velos.App remote Running: /usr/lib/jvm/java/bin/java -client -Ddaemon.name= -Dstorm.options= -Dstorm

问题 We have java app which runs on app server. We have topic topic1 with 5 partitions. We have created 5 consumer threads in a consumer group "consumergroup1". One consumer thread per partition. They are running on appserver server one. Now we bring up one more app server servertwo on another node. We want to run some of consumers in same group "consumergroup1" for topic1 on new server. Basically distribute consumer threads between two app servers for same consumer group. If we bring up consumers

问题 I implemented a bunch of integration tests using EmbededKafka to test one of our Kafka streams application running using spring-kafka framework. The stream application is reading a message from a Kafka topic, it stores it into an internal state store, does some transformation and sends it to another micro service into a requested topic. When the response comes back into the responded topic it retrieves the original message from the state store and depending on some business logic it forwards

问题 I use Spring KafKa anf I set ackMode to MANUAL_IMMEDIATE props.setAckMode(AbstractMessageListenerContainer.AckMode.MANUAL_IMMEDIATE); the scenario is that for some reason my app could not acknowledge ( acknowledgment.acknowledge() ) and just miss it without exception. 1- How can I set consumer retry for missed message 2- How configure to call a function after max retry count that I configured reached 回答1: See the documentation about SeekToCurrentErrorHandlers. When the listener throws an

问题 From https://docs.confluent.io/3.1.1/connect/connect-jdbc/docs/sink_config_options.html#jdbc-sink-configuration-options Specifies how many records to attempt to batch together for insertion into the destination table, when possible. Type: int Default: 3000 Valid Values: [0,…] Importance: medium So, this is from Confluent site. Importance is medium, default is 3000. What if I want the KAFKA changes every 30 secs even if there are say, only 27 KAFKA messages for the topic? What is default