Spring Kafka - How to Retry with @KafkaListener

Question

Question from Twitter:

Just trying to find out a simple example with spring-kafka 2.1.7 that works with a KafkaListener and AckMode.MANUAL_IMMEDIATE , to retry last

Answer 1

it's generally better to ask such questions on Stack Overflow (tagged with spring-kafka.

There are two ways:

• Add a RetryTemplate to the listener container factory - the retries will be performed in memory and you can set backoff properties.
• Add a SeekToCurrentErrorHandler which will re-seek the unprocessed records.

Here is an example:

@SpringBootApplication
public class Twitter1Application {

    public static void main(String[] args) {
        SpringApplication.run(Twitter1Application.class, args);
    }

    boolean fail = true;

    @KafkaListener(id = "foo", topics = "twitter1")
    public void listen(String in, Acknowledgment ack) {
        System.out.println(in);
        if (fail) {
            fail = false;
            throw new RuntimeException("failed");
        }
        ack.acknowledge();
    }

    @Bean
    public ConcurrentKafkaListenerContainerFactory kafkaListenerContainerFactory(
            ConcurrentKafkaListenerContainerFactoryConfigurer configurer,
            ConsumerFactory kafkaConsumerFactory) {
        ConcurrentKafkaListenerContainerFactory factory = new ConcurrentKafkaListenerContainerFactory<>();
        configurer.configure(factory, kafkaConsumerFactory);
        factory.getContainerProperties().setErrorHandler(new SeekToCurrentErrorHandler());
        // or factory.setRetryTemplate(aRetryTemplate);
        // and factory.setRecoveryCallback(aRecoveryCallback);
        return factory;
    }

    @Bean
    public ApplicationRunner runner(KafkaTemplate template) {
        return args -> {
            Thread.sleep(2000);
            template.send("twitter1", "foo");
            template.send("twitter1", "bar");
        };
    }

    @Bean
    public NewTopic topic() {
        return new NewTopic("twitter1", 1, (short) 1);
    }

}

and

spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.consumer.enable-auto-commit=false

spring.kafka.listener.ack-mode=manual-immediate

logging.level.org.springframework.kafka=debug

and

    4.0.0

    com.example
    twitter1
    0.0.1-SNAPSHOT
    jar

    twitter1
    Demo project for Spring Boot

    
        org.springframework.boot
        spring-boot-starter-parent
        2.0.4.RELEASE
         
    

    
        UTF-8
        UTF-8
        1.8
    

    
        
            org.springframework.boot
            spring-boot-starter
        
        
            org.springframework.kafka
            spring-kafka
        

        
            org.springframework.boot
            spring-boot-starter-test
            test
        
    

    
        
            
                org.springframework.boot
                spring-boot-maven-plugin
            
        
    

(Boot 2.0.4 pulls in 2.1.8, which is the current version).

and

foo
2018-08-13 17:36:14.901 ERROR 3945 --- [      foo-0-C-1] essageListenerContainer$ListenerConsumer : Error handler threw an exception

org.springframework.kafka.KafkaException: Seek to current after exception; nested exception is  ...    

2018-08-13 17:36:15.396 DEBUG 3945 --- [      foo-0-C-1] essageListenerContainer$ListenerConsumer : Received: 2 records
foo
2018-08-13 17:36:15.398 DEBUG 3945 --- [      foo-0-C-1] essageListenerContainer$ListenerConsumer : Committing: {twitter1-0=OffsetAndMetadata{offset=5, metadata=''}}
bar
2018-08-13 17:36:15.403 DEBUG 3945 --- [      foo-0-C-1] essageListenerContainer$ListenerConsumer : Committing: {twitter1-0=OffsetAndMetadata{offset=6, metadata=''}}

In the upcoming 2.2 release, the error handler can be configured with a recoverer and standard recoverer is provided to publish the failed record to a dead-letter topic.

Commit here. Docs Here.