Spring @KafkaListener execute and poll records after certain interval

Question

We wanted to consume the records after a certain interval (e.g. every 5 minutes). Consumer properties are standard:

@Bean
public KafkaListenerContainerFactor         


        

Answer 1

If you want to control rate at which Kafka consumer using Spring @KafkaListener, please autowire KafkaListenerEndpointRegistry bean use in following way and access the required MessageListenerContainer. thereafter, you can use the pause() and resume() functionalities to control the required behaviour.

@Autowired
private KafkaListenerEndpointRegistry listener;

@Autowired
private Map<String, Set<String>> getTopicListenerMap(){
    List<String> ids = new ArrayList<>(listener.getListenerContainerIds());
    Map<String, Set<String>> topicListenerMap = new HashMap<>();
    for(String topic: topics){
        topicListenerMap.put(topic, new HashSet<>());
    }
    for(String key: ids){
        for (String topic : listener.getListenerContainer(key).getContainerProperties().getTopics()){
            topicListenerMap.get(topic).add(key);
        }
    }
    return topicListenerMap;
}

@KafkaListener(topics = "topic", containerFactory = "smsListener")
public void listenWithHeaders(@Payload List<String> messageList, @Header(KafkaHeaders.RECEIVED_PARTITION_ID) List<Integer> partitionList,
                              @Header(KafkaHeaders.OFFSET) List<Integer> offsetList) {
    try{
        LOG.info("Received message count: "+(messageList!=null ? messageList.size(): 0)+", offset start: "+offsetList.get(0)+", end: "+offsetList.get(offsetList.size()-1));
        pauseIfRequired(topic);
        for(int i=0; i<messageList.size(); i++){
            // process the messages
        }
    }catch (Exception e){
        LOG.error("", e);
    }finally {
        resumeIfPaused(topic);
    }
}

private void pauseIfRequired(String topic){
    try{ 
        boolean flag = pausingCondition;
        if(flag){
            LOG.info("pausing topic: "+topic);
            for(String listenerKey: getTopicListenerMap().get(topic)){
                listener.getListenerContainer(listenerKey).pause();
            }
            LOG.info("topic paused: "+topic);
        }
    } catch (Exception e){
        LOG.error("", e);
    }
}

private void resumeIfPaused(String topic){
    try {
        for (String listenerKey : getTopicListenerMap().get(topic)) {
            LOG.info("topic: "+topic+", containerPauseRequested: "+listener.getListenerContainer(listenerKey).isPauseRequested());
            if (listener.getListenerContainer(listenerKey).isPauseRequested()) {
                LOG.info("waiting to resume topic: " + topic + ", listener key: " + listenerKey);
                // wait while the condition to resume is fulfilled
                LOG.info("resuming topic: " + topic + ", listener key: " + listenerKey);
                listener.getListenerContainer(listenerKey).resume();
                LOG.info("topic resumed: " + topic + ", listener key: " + listenerKey);
            }
        }
    } catch (Exception e){
        LOG.error("", e);
    }
}

Answer 2

You cannot control the rate at which the consumer polls, the pollTimeout is how long the poll() will wait for new records to arrive. If new records arrive more often, it will not wait that long.

If you wish to control the rate at which you receive records, simply use the DefaultKafkaConsumerFactory to create a consumer and poll it whenever you want.

You can't use that with a @KafkaListener though - you have to deal with the record yourself.

Answer 3

This feature was introduced in 2.3 version.

Starting with version 2.3, the ContainerProperties provides an idleBetweenPolls option to let the main loop in the listener container to sleep between KafkaConsumer.poll() calls. An actual sleep interval is selected as the minimum from the provided option and difference between the max.poll.interval.ms consumer config and the current records batch processing time.

https://docs.spring.io/spring-kafka/reference/html/

KafkaListenerConfig.java

package br.com.sicredi.spi.icom.consumer.config;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.annotation.EnableKafka;
import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;
import org.springframework.kafka.core.ConsumerFactory;
import org.springframework.kafka.core.DefaultKafkaConsumerFactory;
import java.util.HashMap;
import java.util.Map;

@EnableKafka
@Configuration
public class KafkaListenerConfig {

    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, String> concurrentKafkaListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory());
        
        factory.getContainerProperties().setIdleBetweenPolls(100); // 100 miliseconds
        
        return factory;
    }

    private ConsumerFactory<String, String> consumerFactory() {
        return new DefaultKafkaConsumerFactory<>(consumerConfig());
    }

    private Map<String, Object> consumerConfig() {
        Map<String, Object> props = new HashMap<>();
        // ... 
        return props;
    }
}