apache-kafka-streams

问题 I am currently working with Akka Stream Kafka to interact with kafka and I was wonderings what were the differences with Kafka Streams. I know that the Akka based approach implements the reactive specifications and handles back-pressure, functionality that kafka streams seems to be lacking. What would be the advantage of using kafka streams over akka streams kafka? 回答1: Your question is very general, so I'll give a general answer from my point of view. First, I've got two usage scenario:

问题 I am using Kafka streams 2.2.1. I am using suppress to hold back events until a window closes. I am using event time semantics. However, the triggered messages are only triggered once a new message is available on the stream. The following code is extracted to sample the problem: KStream<UUID, String>[] branches = is .branch((key, msg) -> "a".equalsIgnoreCase(msg.split(",")[1]), (key, msg) -> "b".equalsIgnoreCase(msg.split(",")[1]), (key, value) -> true); KStream<UUID, String> sideA =

问题 It appears as though a Kafka Stream with a session window with a grace period and suppression fails to output a final event if there is no constant stream of input records. Context: We are using change data capture (CDC) to monitor changes to a legacy database. When a user makes changes using the UI, a database transaction will change 1..n tables. Each SQL statement results in a Kafka record. These need to be aggregated in order to create one "trigger record" which is used to start an

问题 When defining a topology in kafka streams, a global state store can be added. It will need a source topic as well as a ProcessorSupplier . The processor receive records and could theorically transform them before adding them to the store. But in case of restoration, the records are directly inserted from the source topic (changelog) into the global state store, skipping eventual transformation done in the processor. +-------------+ +-------------+ +---------------+ | | | | | global | |source

问题 When defining a topology in kafka streams, a global state store can be added. It will need a source topic as well as a ProcessorSupplier . The processor receive records and could theorically transform them before adding them to the store. But in case of restoration, the records are directly inserted from the source topic (changelog) into the global state store, skipping eventual transformation done in the processor. +-------------+ +-------------+ +---------------+ | | | | | global | |source

问题 I would like to have Exactly-once semantics, but I don't want to read message with Consumer. I'd rather read messages with Kafka Streams AP. If I add processing.guarantee=exactly_once to Stream configuration, will exactly-once semantics be kept? 回答1: Exactly-once processing is based on a read-process-write pattern. Kafka Streams uses this pattern and thus, if you write a regular Kafka Streams application that writes the result back to a Kafka topic, you will get exactly-once processing

问题 I would like to have Exactly-once semantics, but I don't want to read message with Consumer. I'd rather read messages with Kafka Streams AP. If I add processing.guarantee=exactly_once to Stream configuration, will exactly-once semantics be kept? 回答1: Exactly-once processing is based on a read-process-write pattern. Kafka Streams uses this pattern and thus, if you write a regular Kafka Streams application that writes the result back to a Kafka topic, you will get exactly-once processing

问题 I have a spring boot kafka stream application with 2 topics consider topics A and B.Topic A has 16 partition and Topic B has 1 partition.Consider the application is deployed in 1 instance having num.stream.threads =16. I ran kafka-consumer-groups.bat command to check how the threads are assigned to the partition in group,got the following output.Topic A and B assigned with 16 threads where 14 threads in topic B is idle. kafka-consumer-groups.bat --bootstrap-server 0.0.0.0:9092 --group <topic

问题 I have a spring boot kafka stream application with 2 topics consider topics A and B.Topic A has 16 partition and Topic B has 1 partition.Consider the application is deployed in 1 instance having num.stream.threads =16. I ran kafka-consumer-groups.bat command to check how the threads are assigned to the partition in group,got the following output.Topic A and B assigned with 16 threads where 14 threads in topic B is idle. kafka-consumer-groups.bat --bootstrap-server 0.0.0.0:9092 --group <topic

问题 I have a Kafka streaming application with kafka-streams and kafka-clients both 2.4.0 with the following configs properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, brokers); properties.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, StreamsConfig.EXACTLY_ONCE); brokers= ip1:port1, ip2:port2,ip3:port3, topic partition: 3 topic replication : 3 Scenario 1 : I start only 2 brokers (stream app still contains three ips of broker in broker ip setting) and when i start the my stream app the