apache-kafka-streams

问题 I am new to Kafka and I'm building a starter project using the Twitter API as a data source. I have create a Producer which can query the Twitter API and sends the data to my kafka topic with string serializer for both key and value. My Kafka Stream Application reads this data and does a word count, but also grouping by the date of the tweet. This part is done through a KTable called wordCounts to make use of its upsert functionality. The structure of this KTable is: Key: {word: exampleWord,

问题 Suppose I have different values for a same message key. For example: { userid: 1, email: user123@xyz.com } { userid: 1, email: user456@xyz.com } { userid: 1, email: user789@xyz.com } In this above case I want only the latest value updated by the user, that is, 'user789@xyz.com'. My kafka stream should give me only the third value and not the previous 2 values. 回答1: Since you've not specified a particular client, I'll show you how this can be done with ksqlDB and the newly-added function,

问题 Suppose I have different values for a same message key. For example: { userid: 1, email: user123@xyz.com } { userid: 1, email: user456@xyz.com } { userid: 1, email: user789@xyz.com } In this above case I want only the latest value updated by the user, that is, 'user789@xyz.com'. My kafka stream should give me only the third value and not the previous 2 values. 回答1: Since you've not specified a particular client, I'll show you how this can be done with ksqlDB and the newly-added function,

问题 Suppose I have different values for a same message key. For example: { userid: 1, email: user123@xyz.com } { userid: 1, email: user456@xyz.com } { userid: 1, email: user789@xyz.com } In this above case I want only the latest value updated by the user, that is, 'user789@xyz.com'. My kafka stream should give me only the third value and not the previous 2 values. 回答1: Since you've not specified a particular client, I'll show you how this can be done with ksqlDB and the newly-added function,

问题 Suppose I have different values for a same message key. For example: { userid: 1, email: user123@xyz.com } { userid: 1, email: user456@xyz.com } { userid: 1, email: user789@xyz.com } In this above case I want only the latest value updated by the user, that is, 'user789@xyz.com'. My kafka stream should give me only the third value and not the previous 2 values. 回答1: Since you've not specified a particular client, I'll show you how this can be done with ksqlDB and the newly-added function,

问题 I have a simple streams application takes one topic as input stream and transforms KeyValues to another like: StoreBuilder<KeyValueStore<Long, CategoryDto>> builder = Stores.keyValueStoreBuilder(Stores.inMemoryKeyValueStore(CategoryTransformer.STORE_NAME), Serdes.Long(), CATEGORY_JSON_SERDE); streamsBuilder.addStateStore(builder) .stream(categoryTopic, Consumed.with(Serdes.Long(), CATEGORY_JSON_SERDE)) .transform(CategoryTransformer::new, CategoryTransformer.STORE_NAME); static class

问题 I have a simple streams application takes one topic as input stream and transforms KeyValues to another like: StoreBuilder<KeyValueStore<Long, CategoryDto>> builder = Stores.keyValueStoreBuilder(Stores.inMemoryKeyValueStore(CategoryTransformer.STORE_NAME), Serdes.Long(), CATEGORY_JSON_SERDE); streamsBuilder.addStateStore(builder) .stream(categoryTopic, Consumed.with(Serdes.Long(), CATEGORY_JSON_SERDE)) .transform(CategoryTransformer::new, CategoryTransformer.STORE_NAME); static class

问题 I have a simple streams application takes one topic as input stream and transforms KeyValues to another like: StoreBuilder<KeyValueStore<Long, CategoryDto>> builder = Stores.keyValueStoreBuilder(Stores.inMemoryKeyValueStore(CategoryTransformer.STORE_NAME), Serdes.Long(), CATEGORY_JSON_SERDE); streamsBuilder.addStateStore(builder) .stream(categoryTopic, Consumed.with(Serdes.Long(), CATEGORY_JSON_SERDE)) .transform(CategoryTransformer::new, CategoryTransformer.STORE_NAME); static class

问题 Is there a way to use the same topic as the source for two different processing routines, when using Kafka Streams DSL? StreamsBuilder streamsBuilder = new StreamsBuilder(); // use the topic as a stream streamsBuilder.stream("topic")... // use the same topic as a source for KTable streamsBuilder.table("topic")... return streamsBuilder.build(); Naive implementation from above throws a TopologyException at runtime: Invalid topology: Topic topic has already been registered by another source.

问题 Lets say we have launched 2 Streaming-Tasks at 2 different machines (instances) with following properties :- public final static String applicationID = "StreamsPOC"; public final static String bootstrapServers = "10.21.22.56:9093"; public final static String topicname = "TestTransaction"; public final static String shipmentTopicName = "TestShipment"; public final static String RECORD_COUNT_STORE_NAME = "ProcessorONEStore"; and using these aforesaid properties, here is how stream-task's