Kafka Streams 2.5.0 requires input topic

问题

Starting with Kafka Streams 2.5.0 it seems like a topology must include an input topic. In Kafka 2.4.1 (and earlier) that is not the case.

I have an application where the topology is just creating a few global state stores that read in data from topics written to by other applications.

With Kafka 2.5.0 I get this error:

13:24:27.161 [<redacted>-7cf1b5c9-4a6e-4bf2-9f77-f7f85f2df3bb-StreamThread-1] ERROR o.a.k.s.p.internals.StreamThread - stream-thread [<redacted>-7cf1b5c9-4a6e-4bf2-9f77-f7f85f2df3bb-StreamThread-1] Encountered the following error during processing:
java.lang.IllegalStateException: Consumer is not subscribed to any topics or assigned any partitions
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1228)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1216)
    at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:853)
    at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:753)
    at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:697)
    at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:670)

If I add a dummy input topic (e.g. via streamsBuilder.stream(Pattern.compile("hack"));) the application starts fine.

Is this behavior to be expected or is it an unintentional change in Kafka Streams 2.5.0?

More details: The use case above may seem a bit weird and I would have to agree. The reason for doing it in the first place was a shortcoming of Interactive Queries where for periods of time the application could not answer queries. I see that issue has been fixed in Kafka Streans 2.5.0 via KIP-535 which is great. I hope to look into IQ again later.

回答1:

There was a regression introduced in 2.5.0 when we switched (back) to using collection subscription. A fix was just merged, so you should upgrade to 2.5.1 or 2.6 when they are released.

回答2:

If you don't have any non-global parts of your topology, there's no reason to have any StreamThreads at all. Meaning you can easily work around this by just setting num.threads to zero -- arguably, you should be doing this anyways to avoid unnecessary overhead and group coordination. Setting this to zero by default when a global-only topology is detected is the "fix" anyway, so you don't need to wait for that

回答3:

private void subscribeConsumer() {
    if (builder.usesPatternSubscription()) {
        // this is old behaviour - is there a config that will revert to this??
        consumer.subscribe(builder.sourceTopicPattern(), rebalanceListener);
    } else {
        consumer.subscribe(builder.sourceTopicCollection(), rebalanceListener);
    }
}

回答4:

Have observed same. See change in StreamThread.java. This has been introduced by: https://issues.apache.org/jira/browse/KAFKA-7317

https://github.com/apache/kafka/pull/7969/

Surely there's a config setting to maintain pre 2.5.0 behaviour (i.e. subscribe with consumer.subscribe(builder.sourceTopicPattern(), rebalanceListener)) ??

See snippet...

来源：https://stackoverflow.com/questions/61342530/kafka-streams-2-5-0-requires-input-topic

标签

apache-kafka-streams