Replay Kafka topic from the last successful message

问题

With the standard configuration of a channel on Spring Cloud Stream a message is retried 3 times and then skipped. If the following message processing completes successfully the offset is committed. That means that under transient exceptions messages can be lost.

Can this behavior be changed, so the channel get stuck on a failing message until the transient condition is repaired?

I have tried configuring the retry template, but you have to specify a number of retries, which looks like a useless parameter, as the desired behavior is retrying forever.

Did anyone fall into these troubles? Thank you.

I also have some doubts about how this can interfere with the max.poll.interval.ms property.

回答1:

Disable retry in the binder and use a ListenerContainerCustomizer to add a SeekToCurrentErrorHandler with infinite retries...

@SpringBootApplication
public class So63193500Application {

    public static void main(String[] args) {
        SpringApplication.run(So63193500Application.class, args);
    }

    @Bean
    Consumer<String> input() {
        return str -> {
            System.out.println(str);
            throw new RuntimeException("test");
        };
    }

    @Bean
    ListenerContainerCustomizer<AbstractMessageListenerContainer<?, ?>> customizer() {
        return (container, dest, group) -> {
            if (group.equals("so63193500")) {
                container.setErrorHandler(new SeekToCurrentErrorHandler(
                        new FixedBackOff(5_000, FixedBackOff.UNLIMITED_ATTEMPTS)));
            }
        };
    }

}

spring.cloud.stream.bindings.input-in-0.consumer.max-attempts=1
spring.cloud.stream.bindings.input-in-0.group=so63193500

This causes a seek and won't affect the poll interval as long as the back off interval is not too long.

You can also use an ExponentialBackOff.

来源：https://stackoverflow.com/questions/63193500/replay-kafka-topic-from-the-last-successful-message

标签

Spring

apache-kafka

spring-cloud-stream