spring-kafka

问题 I am following this template for Spring-cloud-stream-kafka but got stuck while making the producer method transactional . I have not used kafka earlier so need help with this in case any configuration changes needed in kafka It works well if no transactional configuration added but when transactional configurations are added it gets timed out at startup - 2020-11-21 15:07:55.349 ERROR 20432 --- [ main] o.s.c.s.b.k.p.KafkaTopicProvisioner : Failed to obtain partition information org.apache

问题 I am using Spring Kafka and one of the topics contains messages with personal data. Is there any way I can configure Spring Kafka to automatically encrypt messages in a Producer/decrypt messages in a consumer or would I have to do it manually? 回答1: There is nothing built into Spring or Kafka (although you can use SSL on the wire to prevent snooping. For application-level encryption/decryption, you would need to implement it. You can separate the concern from your business logic by using a

问题 I implemented a bunch of integration tests using EmbededKafka to test one of our Kafka streams application running using spring-kafka framework. The stream application is reading a message from a Kafka topic, it stores it into an internal state store, does some transformation and sends it to another micro service into a requested topic. When the response comes back into the responded topic it retrieves the original message from the state store and depending on some business logic it forwards

问题 I use Spring KafKa anf I set ackMode to MANUAL_IMMEDIATE props.setAckMode(AbstractMessageListenerContainer.AckMode.MANUAL_IMMEDIATE); the scenario is that for some reason my app could not acknowledge ( acknowledgment.acknowledge() ) and just miss it without exception. 1- How can I set consumer retry for missed message 2- How configure to call a function after max retry count that I configured reached 回答1: See the documentation about SeekToCurrentErrorHandlers. When the listener throws an

问题 I have a simple producer class defined as follows: @Configuration public class MyKafkaProducer { private final static Logger log = LoggerFactory.getLogger(MyKafkaProducer.class); @Value("${my.kafka.producer.topic}") private String topic; @Autowired KafkaTemplate<String, String> kafkaTemplate; public void sendDataToKafka(@RequestParam String data) { ListenableFuture<SendResult<String, String>> listenableFuture = kafkaTemplate.send(topic, data); listenableFuture.addCallback(new

问题 I'm thinking about the following use-case and would like to validate if this approach is conceptually valid. The goal is to expose a long-running Server-Sent-Event (SSE) endpoint in Spring replaying the same Kafka topic for each incoming connection (with some user-specific filtering). The SSE is exposed in this way: @GetMapping("/sse") public SseEmitter sse() { SseEmitter sseEmitter = new SseEmitter(); Executors .newSingleThreadExecutor() .execute(() -> dummyDataProducer.generate() // kafka

问题 I am using RequestReplyFuture<String, String, List> to mapped the response to List, the result is something like below @Service public class ProductProducer implements IProductProducer{ private final ReplyingKafkaTemplate<String, String, List<Product>> _replyTemplate; private static final Logger LOG = LoggerFactory.getLogger(ProductProducer.class); public ProductProducer(ReplyingKafkaTemplate<String, String, List<Product>> replyTemplate) { this._replyTemplate = replyTemplate; } @Override

问题 Current setup - I am streaming data from 16 single partitioned topics and doing KTable-KTable joins and sending an output with aggregated data from all streams. I am also materializing each KTable to local state-store. Scenarios - When I tried running two app instances, I was expecting it kafka streams to run on single instance but for some reason it ran on other instance too. Looks like it can created stream task on other app instance during kafka streams failure on instance#1 during to some

问题 I am trying to schedule my consumption process from a single partition topic. I can start it using endpointlistenerregistry.start() but I want to stop it after I have consumed all the messages in current partition i.e. when I reach to last offset in current partition. Production into the topic is done after I have finished the consumption and close it. How should I achieve the assurance that I have read all the messages till the time I started scheduler and stop my consumer ? I am using

问题 I am trying to schedule my consumption process from a single partition topic. I can start it using endpointlistenerregistry.start() but I want to stop it after I have consumed all the messages in current partition i.e. when I reach to last offset in current partition. Production into the topic is done after I have finished the consumption and close it. How should I achieve the assurance that I have read all the messages till the time I started scheduler and stop my consumer ? I am using