How to run kafka streams effectively with single app instance and single topic partitions?

问题

Current setup - I am streaming data from 16 single partitioned topics and doing KTable-KTable joins and sending an output with aggregated data from all streams. I am also materializing each KTable to local state-store.

Scenarios -

• When I tried running two app instances, I was expecting it kafka streams to run on single instance but for some reason it ran on other instance too. Looks like it can created stream task on other app instance during kafka streams failure on instance#1 during to some streams exception. Big backlog on topics was cleared but when i query aggregated data state-store through interactiveQueryService, I see few streams data missing. Looks like it created state-store for few streams on instance#2. Not sure though.

• When I try to run on single instance with this heavy backlog, I am seeing all kind of exception - Timeout exception, RecordTooLarge exception. For recordtoolarge exception, i have implemented productionexceptionhandler to catch exception and continue. But looks it is timing out and not KafkaStreams keeps crashing -

  1. org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. You can increase producer parameter
     retries and retry.backoff.ms to avoid this error.
  2. Expiring 5 record(s) for partition-0: 381960 ms has passed since last append; No more records will be sent and no more offsets will be recorded for this task.
  3. Abort sending since an error caught with a previous record (key 1 value [B@2 timestamp 1548700011925) to topic abc-xyz-store-changelog due to org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

current configurations -

• retry.backoff.ms: 5000 request.timeout.ms: 300000 retries: 5 commit.interval.ms: 1000 PROCESSING_GUARANTEE:at_least_once

All other values are default like -

• MAX_POLL_INTERVAL_MS:2147483647 MAX_POLL_RECORDS:1000 BATCH_SIZE:16384 NUM_STREAM_THREADS:1 LINGER_MS:100 MAX_BLOCK_MS:60000 MAX_IN_FLIGHT_REQ_PER_CONN:5

I want to run the app on single instance without crashing even if it process slowly to make sure it clears backlog.

来源：https://stackoverflow.com/questions/54416023/how-to-run-kafka-streams-effectively-with-single-app-instance-and-single-topic-p