Retrieve a message by knowing its partition and offset in Kafka

筅森魡賤 提交于 2020-02-24 14:29:31

问题


I'm working on Kafka 0.9. I'm wondering if there is any approach to retrieve a message, which has been processed, from its topic by knowing the partition and offset. For example, the consumer is currently consuming the message at partition 1 and offset 10. And I want to get the message at the same partition and offset 5.

One way that I can think of is to reset the offset to 5 and consume one single message. But the poll() method can only return a batch of messages. So I have to take the first message and disregard the others. After processing the message, the offset is reset back.

I think this will work. But still want to know if there is any other elegant way of doing it.


回答1:


You should be able to use the "seek" method to read the message from the offset you require.

Take a look at the "Controlling the Consumer's Position" https://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html




回答2:


Kafka is designed to read long stripes of data off of the disk without moving the disk heads around -- in other words, it is optimized to use linear reads. It seems inefficient to disregard a whole chunk of data you had to read off of disk (and possibly serve over the network) but it is actually a lot more inefficient to make the disk head jump around a lot. Check out Kafka's design philosophy, and about it's use of disks, here.

In other words, your approach probably works. But you are thinking more like the way someone uses a relational database, not a messaging system.



来源:https://stackoverflow.com/questions/36435230/retrieve-a-message-by-knowing-its-partition-and-offset-in-kafka

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!