How does an offset expire for an Apache Kafka consumer group?

Question

I was making some tests on an old topic when I noticed some strange behaviours. Reading Kafka\'s log I noticed this \"removed 8 expired offsets\" message:

[         


        

Answer 1

Check my answer here. You should not forget about file rolling. It impacts offset files removal.

Answer 2

Kafka, by default deletes committed offsets after a configurable period of time. See parameter offsets.retention.minutes. Ie, if a consumer group is inactive (ie, does not commit any offsets) for this amount of time, the offsets get deleted. Thus, even if the consumer is running, if it does not commit offsets for some partitions, those offsets are subject to offset.retention.minutes.

If you start a consumer, the following happens:

1. look for a (valid) committed offset (for the consumer group)
   1. if valid offset is found, resume from there
   2. if no valid offset is found, reset offset according to auto.offset.reset parameter

Thus, if your offsets got deleted and auto.offset.reset = latest, you consumer will not poll anything until new data is added to the topic. If auto.offset.reset = earliest it should consume the whole topic.

See this JIRA for a discussion about this https://issues.apache.org/jira/browse/KAFKA-3806 and https://issues.apache.org/jira/browse/KAFKA-4682