apache-kafka

问题 I'm using Kafka and Kafka Streams as part of Spring Cloud Stream. The data that is flowing in my Kafka Streams app is being aggregated and materialized by certain time windows: Materialized<String, ErrorScore, WindowStore<Bytes, byte[]>> oneHour = Materialized.as("one-hour-store"); oneHour.withLoggingEnabled(topicConfig); events .map(getStringSensorMeasurementKeyValueKeyValueMapper()) .groupByKey() .windowedBy(TimeWindows.of(oneHourStore.getTimeUnit())) .reduce((aggValue, newValue) ->

问题 I'm working on a Kafka streams application written in Kotlin, and I'm seeing some bizarre behavior with a join. At a high level, I'm streaming two topics with different keys. However, I can rekey one of the messages so that they keys line up. After I do this though, the subsequent join is not fired. Below I have supplied the simplified code (with irrelevant portions elided and replaced with comments) val builder = KStreamBuilder() val joinWindow = JoinWindows.of(/* 30 days */).until(/* 365

问题 I'm working on a Kafka streams application written in Kotlin, and I'm seeing some bizarre behavior with a join. At a high level, I'm streaming two topics with different keys. However, I can rekey one of the messages so that they keys line up. After I do this though, the subsequent join is not fired. Below I have supplied the simplified code (with irrelevant portions elided and replaced with comments) val builder = KStreamBuilder() val joinWindow = JoinWindows.of(/* 30 days */).until(/* 365

问题 I'm working on a Kafka streams application written in Kotlin, and I'm seeing some bizarre behavior with a join. At a high level, I'm streaming two topics with different keys. However, I can rekey one of the messages so that they keys line up. After I do this though, the subsequent join is not fired. Below I have supplied the simplified code (with irrelevant portions elided and replaced with comments) val builder = KStreamBuilder() val joinWindow = JoinWindows.of(/* 30 days */).until(/* 365

问题 I see from the logs that exact same message is consumed by the 665 times. Why does this happen? I also see this in the logs Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured session.timeout.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session

问题 I see from the logs that exact same message is consumed by the 665 times. Why does this happen? I also see this in the logs Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured session.timeout.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session

问题 I am looking for some suggestion on Kafka broker auto scaling up and down based on the load. Let us say we have an e-commerce site and we are capturing certain activities or events and these events are send to Kafka. Since during the peak hours/days the site traffic will be more so having the ideal kafka cluster with fixed number of brokers always is not a good idea so we want to scale it up the number of brokers when site traffic is more and scale it down the number of brokers when traffic

问题 I am looking for some suggestion on Kafka broker auto scaling up and down based on the load. Let us say we have an e-commerce site and we are capturing certain activities or events and these events are send to Kafka. Since during the peak hours/days the site traffic will be more so having the ideal kafka cluster with fixed number of brokers always is not a good idea so we want to scale it up the number of brokers when site traffic is more and scale it down the number of brokers when traffic

问题 I'm trying to make a simple chat app using tornado based on this But also I want to use kafka to store the messages. How can I do that? Now, I used this to make a consumer and somehow it's working but it's only printing on the console and I need the messages to show on the webpage, like the tornade app, only it's saved in kafka. Here's my app.py code as of now #!/usr/bin/env python # # Copyright 2009 Facebook # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not

问题 I'm trying to make a simple chat app using tornado based on this But also I want to use kafka to store the messages. How can I do that? Now, I used this to make a consumer and somehow it's working but it's only printing on the console and I need the messages to show on the webpage, like the tornade app, only it's saved in kafka. Here's my app.py code as of now #!/usr/bin/env python # # Copyright 2009 Facebook # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not