apache-kafka-streams

问题 i am using following log4j.properties log4j.rootLogger=DEBUG, stdout log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.Target=System.out log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n I wanted to disable log messages for kafka only. where as display my log messages being logged. 回答1: You need to set the log level to OFF by adding this line: log4j.logger.org

问题 i am using following log4j.properties log4j.rootLogger=DEBUG, stdout log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.Target=System.out log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n I wanted to disable log messages for kafka only. where as display my log messages being logged. 回答1: You need to set the log level to OFF by adding this line: log4j.logger.org

问题 I am trying to create a Kafka topology and break it down into more readable. I have a stream that I group by keys, and then I am trying to window it like so: SessionWindowedKStream<byte[], byte[]> windowedTable = groupedStream.windowedBy(SessionWindows.with(Duration.ofSeconds(config.joinWindowSeconds)).grace(Duration.ZERO)); KTable<Windowed<byte[]>, byte[]> mergedTable = windowedTable .reduce((aggregateValue, newValue) -> { try { Map<String, String> recentMap = MAPPER.readValue(new String

问题 I am trying to create a Kafka topology and break it down into more readable. I have a stream that I group by keys, and then I am trying to window it like so: SessionWindowedKStream<byte[], byte[]> windowedTable = groupedStream.windowedBy(SessionWindows.with(Duration.ofSeconds(config.joinWindowSeconds)).grace(Duration.ZERO)); KTable<Windowed<byte[]>, byte[]> mergedTable = windowedTable .reduce((aggregateValue, newValue) -> { try { Map<String, String> recentMap = MAPPER.readValue(new String

问题 I am trying to create a Kafka topology and break it down into more readable. I have a stream that I group by keys, and then I am trying to window it like so: SessionWindowedKStream<byte[], byte[]> windowedTable = groupedStream.windowedBy(SessionWindows.with(Duration.ofSeconds(config.joinWindowSeconds)).grace(Duration.ZERO)); KTable<Windowed<byte[]>, byte[]> mergedTable = windowedTable .reduce((aggregateValue, newValue) -> { try { Map<String, String> recentMap = MAPPER.readValue(new String

问题 When our kafka stream application attempts to recover state from the changelog topic our rocksdb state store directory continually grows (10GB+) until we run out of disk space and never actually recovers. How I can reproduce. I start up our application with a brand new changelog topic. I push a few hundred thousand records through. I note my RocksDb state store is around 100mb. I gracefully shutdown the application and restart it. I see the restore consumers logging and stating they are

问题 I have the following properties for my Spring Cloud Stream application that uses Kafka Streams Binder: spring.cloud.stream.bindings: windowStream-in-0: destination: input windowStream-out-0: destination: window1 hint1Stream-in-0: destination: window1 hint1Stream-out-0: destination: hints realityStream-in-0: destination: input realityStream-in-1: destination: window1 consumer: timestampExtractorBeanName: anotherTimestampExtractor realityStream-out-0: destination: hints countStream-in-0:

问题 I have the following properties for my Spring Cloud Stream application that uses Kafka Streams Binder: spring.cloud.stream.bindings: windowStream-in-0: destination: input windowStream-out-0: destination: window1 hint1Stream-in-0: destination: window1 hint1Stream-out-0: destination: hints realityStream-in-0: destination: input realityStream-in-1: destination: window1 consumer: timestampExtractorBeanName: anotherTimestampExtractor realityStream-out-0: destination: hints countStream-in-0:

问题 I am trying to understand Stateful Stream processor . As I understand in this type of stream-processor, it maintains some sort of state using State Store . I came to know, one of the ways to implement State Store is using RocksDB . Assuming the following topology (and only one processor being stateful ) A->B->C ; processor B as stateful with local state store and changelog enabled . I am using low level API. Assuming the sp listens on a single kafka topic, say topic-1 with 10 partitions. I

问题 I read lot about global state store that it does not create change-topic topic for restore instead it use the source topic as restore. i am create custom key and store the data in global state store, but after restart it will gone because global store on restore will directly take data from source topic and bypass the processor. my input topic has above data. { "id": "user-12345", "user_client": [ "clientid-1", "clientid-2" ] } i am maintaining two state store as follow: id ->record (record