Spring-Kafka Producer Retry when all brokers are down

问题

I am using Springboot 2.3.5.RELEASE along with spring-kafka 2.6.3. I am trying do a simple Kafka Producer Retry POC which should result in a producer retrying when the broker is down or if there is an exception thrown before a message is sent to broker.
The below producer config is for Idempotent producer with retries enabled.

// Producer configuration
@Bean
public Map<String, Object> producerConfigs() {
   Map<String, Object> props = new HashMap<>();
   props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress);
   props.put(JsonDeserializer.TRUSTED_PACKAGES, "*");
   props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
   props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, "3");
   props.put(ProducerConfig.RETRIES_CONFIG, "10");
   props.put(ProducerConfig.ACKS_CONFIG, "all");
}
@Bean
public ProducerFactory<String, Object> producerFactory() {
    return new DefaultKafkaProducerFactory<>(producerConfigs());
}
@Bean
public KafkaTemplate<String, Object> kafkaTemplate() {
    return new KafkaTemplate<>(producerFactory());
}
... other configs

Iam aware that Kafka Producer Retries work only if there is a transient error in broker sending ack to producer. So, if the broker is down even before the message is sent then the above retry does not work. Therefore I tried to introduce @Retry with spring-kafka, unfortunately I am unable to get this to work either. Does anybody know how to solve this as this seems to be a common usecase like if the broker goes down or if there is a network glitch I don't want my producers to stop working

@Transaction
@Retryable(value = Exception.class, maxAttemptsExpression = "10",
        backoff = @Backoff(delayExpression = "1000"))
public void sendTestMessage(String name, String number) throws RuntimeException {
    TestObject testObj = new TestObject(name + ": " + String.valueOf(number),
            Double.valueOf(Math.random() * 10000).longValue(),
            "Blah" + String.valueOf(Math.random() * 10));
    log.info("Sending sendTestMessage {}", name); 
    throwSomeException();
    // sends the message
    kafkaTemplate.send(MyConstants.SomeTOPIC, testObj);     
}  
// Throwing mock exception
public void throwSomeException() throws RuntimeException {
        log.info("Throwing mock exception {}");
        throw new RuntimeException("Mock Exception thrown while sending the 
        sendTestMessage method");
}   
@Recover
public void recover(Exception e, TestObject testObj) { 
    log.info("Recovered exception thrown  mock exception {} with payload as {}", e, testObj);
}

Is there anyway a Retry could work for the above. I find it hard to believe that such a functionality is not available inbuilt by default.

回答1:

I solved the issue and I am posting this for benefit of others. I was missing the below in my config class

@Configuration
@EnableRetry <-- this was missing
public class ProducerConfig {
  ...

After adding the above it is working fine.

来源：https://stackoverflow.com/questions/65247844/spring-kafka-producer-retry-when-all-brokers-are-down

标签

spring-boot

spring-kafka