Kafka Configuration for only seeing last 5 minutes of data

若如初见. 提交于 2019-12-02 10:22:58

In Kafka server.properties there's a setting called log.segment.bytes, which is set to 1GB by default. Once a log segment has reached 1GB, it is closed, and only after that the retention kicks in. E.g. if you are producing 100MB of message per day, and your retention is 1 week, you'd actually retain the data for around 17 days before it gets deleted. That's because the log segment will take 10 days to be full (1GB) and from that time retention will kick in. In your case, I'm assuming you haven't changed the value for log.segment.bytes, but your retention is very low. So, it won't be able to clean up the data as the log segment is not yet closed.

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!