apache-kafka-streams

问题 I'm working on a Kafka streams application written in Kotlin, and I'm seeing some bizarre behavior with a join. At a high level, I'm streaming two topics with different keys. However, I can rekey one of the messages so that they keys line up. After I do this though, the subsequent join is not fired. Below I have supplied the simplified code (with irrelevant portions elided and replaced with comments) val builder = KStreamBuilder() val joinWindow = JoinWindows.of(/* 30 days */).until(/* 365

问题 I have a single source CSV file containing records of different sizes that pushes every record into one source topic. I want to split the records into different KStreams/KTables from that source topic. I have a pipeline for one table load, where I am pushing the record from the source topic into stream1 in delimited format and then pushing the records into another stream in AVRO format which is then pushed into JDBC sink connector that pushes the record into MySQL database. The pipeline needs

问题 I have a single source CSV file containing records of different sizes that pushes every record into one source topic. I want to split the records into different KStreams/KTables from that source topic. I have a pipeline for one table load, where I am pushing the record from the source topic into stream1 in delimited format and then pushing the records into another stream in AVRO format which is then pushed into JDBC sink connector that pushes the record into MySQL database. The pipeline needs

问题 We are running kafka streams applications and frequency running into off heap memory issues. Our applications are deployed and kubernetes PODs and they keep on restarting. I am doing some investigation and found that we can limit the off heap memory by implementing RocksDBConfigSetter as shown in following example. public static class BoundedMemoryRocksDBConfig implements RocksDBConfigSetter { // See #1 below private static org.rocksdb.Cache cache = new org.rocksdb.LRUCache(TOTAL_OFF_HEAP

问题 I'm having problems joining 2 kafka streams extracting the date from the fields of my event. The join is working fine when I do not define a custom TimeStampExtractor but when I do the join does not work anymore. My topology is quite simple: val builder = new StreamsBuilder() val couponConsumedWith = Consumed.`with`(Serdes.String(), getAvroCouponSerde(schemaRegistryHost, schemaRegistryPort)) val couponStream: KStream[String, Coupon] = builder.stream(couponInputTopic, couponConsumedWith) val

问题 We are using kafka-streams 2.3.1 and I've just noticed that if broker is down, the streams app seems to be content to try to keep trying connecting forever. new KafkaStreams(createTopology(), properties()).start() o.apache.kafka.clients.NetworkClient - [AdminClient clientId=test] Connection to node -1 (broker/127.0.0.1:9092) could not be established. Broker may not be available. The streams state is REBALANCING while this is going on so there's no good way to determine if the connection is

问题 I am trying to access the inMemoryStore that I am creating with in the same java program. But returning a exception as "Exception in thread "main" org.apache.kafka.streams.errors.InvalidStateStoreException: The state store, storeName, may have migrated to another instance." When I am using the persistentKeyValueStore it is working fine and able to create the store and return the values. package com.bakdata.streams_store.demo; import java.util.Collection; import java.util.Properties; import

问题 I have a Kafka Streams application, which shuts down without any proper logging even at the debug level - 2020-12-18 14:25:36:875 +0000 [Thread-7] INFO o.apache.kafka.streams.KafkaStreams:? - stream-client [trinity-client-pandprat-estestes5-null-b9346744-6bb4-464d-aeaa-9311ab16ce2c] State transition from REBALANCING to PENDING_SHUTDOWN 2020-12-18 14:25:36:973 +0000 [kafka-streams-close-thread] INFO o.a.k.s.p.internals.StreamThread:? - stream-thread [trinity-client-pandprat-estestes5-null

问题 Is it possible to log inbound messages from stream function? Is there some kind of interceptor that allows me to do that? 来源： https://stackoverflow.com/questions/65683790/spring-cloud-stream-kafka-streams-how-to-log-incomming-messages-in-stream

问题 Starting up a Stream application (using Kafka Streams) fails with the "java.lang.IllegalStateException: This should not happen as headers() should only be called while a record is processed" This seems to only happen when I start up the application if there is already data in the topic. If the topic is empty and I start pushing data to it, all is fine. Would someone know why this would happen? Thanks This should not happen as headers() should only be called while a record is processed java