apache-kafka-streams

问题 I have an AWS infrastructure of 1 Kafka cluster on 3 docker containers, running on ECS, using EFS as storage service (for simplicity). 1 Kafka Streams application cluster, on 3 containers. There is a source topic "events" with 16 partitions, replication 2. A PAPI topology processor "stream-processor" produces output to some other topics and uses 3 state stores. I can see via Kafka Manager that data is consumed, and output is produced to these other output topics. Apparently it works (though

问题 I need a stream to group values from topic A, send the grouped values to topic B, and send the sum of those grouped values over a tumbling time window to topic C. Is it possible to do that in Kafka? Or can a stream both read and write from one topic only? 回答1: Yes. For example you can split a input stream and write to different topics: KStream stream = ... KStream[] splitStream = stream.branch(...); splitStream[0].to("topic-1"); splitStream[1].to("topic-2"); You can also use a more dynamic

问题 We noticed on our kafka broker logs that our stream applications are trying to describe topics that are not part of the topology. It looks like it is periodically happening every 10 minutes. Why is this occurring? Why does streams need to know about topics that are not part of the topology? Anyway to stop this? Since the stream app does not have access to describe those topics we are getting a bunch of error logs. We are using version 1.1.1 回答1: It's a know issue, that will get fixed in the 2

问题 We noticed on our kafka broker logs that our stream applications are trying to describe topics that are not part of the topology. It looks like it is periodically happening every 10 minutes. Why is this occurring? Why does streams need to know about topics that are not part of the topology? Anyway to stop this? Since the stream app does not have access to describe those topics we are getting a bunch of error logs. We are using version 1.1.1 回答1: It's a know issue, that will get fixed in the 2

问题 I need to create a state store with String key HashMap as value. I tried the below two methods. // First method StateStoreSupplier avgStoreNew = Stores.create("AvgsNew") .withKeys(Serdes.String()) .withValues(HashMap.class) .persistent() .build(); // Second method HashMap<String ,Double> h = new HashMap<String ,Double>(); StateStoreSupplier avgStore1 = Stores.create("Avgs") .withKeys(Serdes.String()) .withValues(Serdes.serdeFrom(h.getClass())) .persistent() .build(); The code compiles fine

问题 Since I am working with Json values I haven't set up default serdes. I process a KStream, consuming it with the necessary spring and product (json) serdes, but the next step (map operation) fails: val props = Properties() props[StreamsConfig.APPLICATION_ID_CONFIG] = applicationName props[StreamsConfig.BOOTSTRAP_SERVERS_CONFIG] = kafkaBootstrapServers val productSerde: Serde<Product> = Serdes.serdeFrom(JsonPojoSerializer<Product>(), JsonPojoDeserializer(Product::class.java)) builder.stream

问题 I want to join a KStream with a KTable. Both have a different key but are co-partitioned using a custom partitioner. However, the join does not produce and results. The KStream has the following structure - key: House - Group - value: User The KTable has the following structure - key: User - Group - value: Address To make sure every insert both topics are processed in insertion order, I'm using a custom Partitioner where I'm partitioning both topics using the Group part of each key. I want to

问题 I want to join a KStream with a KTable. Both have a different key but are co-partitioned using a custom partitioner. However, the join does not produce and results. The KStream has the following structure - key: House - Group - value: User The KTable has the following structure - key: User - Group - value: Address To make sure every insert both topics are processed in insertion order, I'm using a custom Partitioner where I'm partitioning both topics using the Group part of each key. I want to

问题 I am recently trying to play with kafka streaming for some sensitive data processing. The goal I wish to achieve is that while the sensitive data is encrypted, the power of microservice architecture is not jeopardised, i.e., losely coupled services and stream data processing. My question is, in kafka streaming, is it possible that I decrypt an incoming message using one key and encrypt it again with another key? I kind of got a plan, but as I am not familiar with kafka streaming, I can not

问题 I'm trying to create a leftJoin in Kafka Streams which works fine for about 10 Records and then it crashes with an exception caused by a NullPointerException with such code: private static KafkaStreams getKafkaStreams() { StreamsConfig streamsConfig = new StreamsConfig(getProperties()); KStreamBuilder builder = new KStreamBuilder(); KTable<String, Verkaeufer> umsatzTable = builder.table(Serdes.String(), EventstreamSerde.Verkaeufer(), CommonUtilsConstants.TOPIC_VERKAEUFER_STAMMDATEN); KStream