apache-kafka-streams

问题 I'm using Kafka Streams that call external systems during map/foreach . Is there any timeout on how long map or foreach can take? Are there any caveats to blocking for a long time (say hours)? 回答1: There is no timeout applied to map/foreach . However, as Kafka Streams uses KafkaConsumer and KafkaProducer internally, all their timeouts apply (eg. max.poll.interval.ms ). You can of course configure them accordingly, but it's not recommended to do long blocking calls to external systems. It

问题 I'm using Kafka Streams that call external systems during map/foreach . Is there any timeout on how long map or foreach can take? Are there any caveats to blocking for a long time (say hours)? 回答1: There is no timeout applied to map/foreach . However, as Kafka Streams uses KafkaConsumer and KafkaProducer internally, all their timeouts apply (eg. max.poll.interval.ms ). You can of course configure them accordingly, but it's not recommended to do long blocking calls to external systems. It

问题 Kafka version: 1.0.0 Let's say the stream application uses low level processor API which maintains the state and reads from a topic with 10 partitions. Please clarify if the internal topic is expected to be created with the same number of partitions OR is it per the broker default. If it's the later, if we need to increase the partitions of the internal topic, is there any option? 回答1: Kafka Streams will create the topic for you. And yes, it will create it with the same number of partitions

问题 Why must a global table not have logging enabled in kafka? code: if (loggingEnabled) { throw new TopologyException("StateStore " + storeName + " for global table must not have logging enabled."); } When I am restarting my kafka application all the datastore in globalstore is getting clean. How could I can made it tolerance. 回答1: GlobalKTable s use their input topic to guarantee fault-tolerance. Thus, I am not sure what you mean by When I am restarting my kafka application all the datastore in

问题 Background Several machines generate events. These events get sent to our Kafka cluster, where each machine has its own topic (app.machine-events. machine-name ). Because order is important on a per-machine basis, and partition-size is not an issue for now, all topics consist of a single partition. Therefore N topics also means N partitions, currently. The consuming/processing app makes use of kafka-streams, which we've given the StreamsConfig.APPLICATION_ID_CONFIG / "application.id" 'machine

问题 Background Several machines generate events. These events get sent to our Kafka cluster, where each machine has its own topic (app.machine-events. machine-name ). Because order is important on a per-machine basis, and partition-size is not an issue for now, all topics consist of a single partition. Therefore N topics also means N partitions, currently. The consuming/processing app makes use of kafka-streams, which we've given the StreamsConfig.APPLICATION_ID_CONFIG / "application.id" 'machine

问题 Background Several machines generate events. These events get sent to our Kafka cluster, where each machine has its own topic (app.machine-events. machine-name ). Because order is important on a per-machine basis, and partition-size is not an issue for now, all topics consist of a single partition. Therefore N topics also means N partitions, currently. The consuming/processing app makes use of kafka-streams, which we've given the StreamsConfig.APPLICATION_ID_CONFIG / "application.id" 'machine

问题 I have a custom Transformer implementation in the end of my kafka-streams DSL, with a persistent changelog KeyValueStore bound to it. Since few weeks I have been putting way too much data in the store. Now whenever I load the application it eats up way too much RAM. However, the application itself is just a prototype so I don't mind clearing the store entirely. I could rename the kafka.application.id and the state-store-name but that's a temporary workaround(and the corresponding data/topics

问题 I want to dynamically generate some kafka stream code when the program is running， but I get an exception when I compile the following code for kafka stream with scala toolbox eval： val toolbox = runtimeMirror(getClass.getClassLoader).mkToolBox() val code = """ |import org.apache.kafka.streams.scala.kstream.KStream |import org.apache.kafka.streams.scala.StreamsBuilder | |class ClassA { | def createStream(): KStream[String, String] = { | import org.apache.kafka.streams.scala

问题 I want to dynamically generate some kafka stream code when the program is running， but I get an exception when I compile the following code for kafka stream with scala toolbox eval： val toolbox = runtimeMirror(getClass.getClassLoader).mkToolBox() val code = """ |import org.apache.kafka.streams.scala.kstream.KStream |import org.apache.kafka.streams.scala.StreamsBuilder | |class ClassA { | def createStream(): KStream[String, String] = { | import org.apache.kafka.streams.scala