Kafka Consumer offset commit check to avoid committing smaller offsets

问题

We assume that we have a consumer that sends a request to commit offset 10. If there is a communication problem and the broker didn't get the request and of course didn't respond. After that we have another consumer process another batch and successfully committed offset 20.

Q: I want to know if there is a way or property to handle so we can check if the previous offset in the log are committed or not before committing in our case the offset 20?

回答1:

The scenario you are describing can only happen when using asynchronous commits.

Keep in mind that one particular TopicPartition can only be consumed by a single consumer within the same ConsumerGroup. If you have two consumers reading the same TopicPartition it is only possible

if they have different ConsumerGroups, or
if they have the same ConsumerGroup and a Rebalance happens. But still, only one consumer will read that TopicPartition at a time, never both in parallel.

Case #1 is pretty clear: If they have different ConsumerGroups they consume the partition in parallel and independently. Also their comitted offsets are managed separately.

Case #2: If the first consumer fails to commit offset 10 because the consumer failed/died and is not recovering a consumer Rebalance will happen and another active consumer will pick up that partition. As the offset 10 was not committed, the new consumer will start reading again offset 10 before jumping to the next batch and possibly commit offset 20. This leads to "at-least-once" semantics and could lead to duplicates.

Now, coming to the only scenario where you could commit a smaller offset after committing a higher offset. As said in the beginning, this could indeed happen if you asynchronously commit the offsets (using commitAsync). Imagine the following scenario, ordered by time:

Consumer reads offset 0 (background thread tries to commit offset 0)
committing offset 0 succeeded
Consumer reads offset 1 (background thread tries to commit offset 1)
committing offset 1 failed, try again later
Consumer reads offset 2 (background thread tries to commit offset 2)
committing offset 2 succeeded
Now, what to to with (re-trying committing offset 1?)

If you let the retrying mechanism to commit offset 1 again, it looks like your consumer has only committed up until the offset 1. This is because the information for each Consumer group on the latest offset par TopicPartition is stored in the internal compacted Kafka topic __consumer_offsets which is meant to store only the latest value (in our case: offset 1) for our Consumer Group.

In the book "Kafka - The Definitive Guide", there is a hint on how to mitigate this problem:

Retrying Async Commits: A simple pattern to get commit order right for asynchronous retries is to use a monotonically increasing sequence number. Increase the sequence number every time you commit and add the sequence number at the time of the commit to the commitAsync callback. When you’re getting ready to send a retry, check if the commit sequence number the callback got is equal to the instance variable; if it is, there was no newer commit and it is safe to retry. If the instance sequence number is higher, don’t retry because a newer commit was already sent.

As an example, you can see an implementation of this idea in Scala below:

import java.util._
import java.time.Duration
import org.apache.kafka.clients.consumer.{ConsumerConfig, ConsumerRecord, KafkaConsumer, OffsetAndMetadata, OffsetCommitCallback}
import org.apache.kafka.common.{KafkaException, TopicPartition}
import collection.JavaConverters._

object AsyncCommitWithCallback extends App {

  // define topic
  val topic = "myOutputTopic"

  // set properties
  val props = new Properties()
  props.put(ConsumerConfig.GROUP_ID_CONFIG, "AsyncCommitter5")
  props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092")
  // [set more properties...]
  

  // create KafkaConsumer and subscribe
  val consumer = new KafkaConsumer[String, String](props)
  consumer.subscribe(List(topic).asJavaCollection)

  // initialize global counter
  val atomicLong = new AtomicLong(0)

  // consume message
  try {
    while(true) {
      val records = consumer.poll(Duration.ofMillis(1)).asScala

      if(records.nonEmpty) {
        for (data <- records) {
          // do something with the records
        }
        consumer.commitAsync(new KeepOrderAsyncCommit)
      }

    }
  } catch {
    case ex: KafkaException => ex.printStackTrace()
  } finally {
    consumer.commitSync()
    consumer.close()
  }


  class KeepOrderAsyncCommit extends OffsetCommitCallback {
    // keeping position of this callback instance
    val position = atomicLong.incrementAndGet()

    override def onComplete(offsets: util.Map[TopicPartition, OffsetAndMetadata], exception: Exception): Unit = {
      // retrying only if no other commit incremented the global counter
      if(exception != null){
        if(position == atomicLong.get) {
          consumer.commitAsync(this)
        }
      }
    }
  }

}

来源：https://stackoverflow.com/questions/64191835/kafka-consumer-offset-commit-check-to-avoid-committing-smaller-offsets

标签

apache-kafka

kafka-consumer-api