问题

I'm using Apache Kafka API and trying to get only one message at a time. I'm only writing to one topic. I can send and receive messages by having a pop UI screen with a textbox. I input a string in the textbox and click "send." I can send as many messages as I want. Let's say I send 3 messages and my 3 messages were "hi," "lol," "bye." There is also a "receive" button. Right now, using the traditional code found in TutorialsPoint, I get all 3 messages (hi, lol, bye) at once printed on the console when I click on the receive button. However, I want only want one message to be printed at a time when I click "receive" on the UI. For example, the first time I hit the receive button, it would print "hi," the second time would be "lol," and the third time would be "bye." I am new to Kafka and am confused on how to do this. I tried removing both the loops from the code so it just has

ConsumerRecords<String, String> records = consumer.poll(100);
System.out.printf(records.iterator().next().value());

If I just have those 2 lines of code, the first time I hit the receive button, it would print "hi" but the second time I press it, get the message "attempt to heartbeat failed since group is rebalancing kafka." I'm getting errors when I set the max.poll.records = 1 too as I want all my messages eventually but just one of them needs to be logged to the console when the receive button is pressed. The next time, the next message in the topic not logged would be logged.

Hope that makes sense! Appreciate any help! Thanks in advance!

EDIT: New code after including queues and also so we can alternate between sending and receiving messages & update the queue whenever there is a new message:

        if (payloadQueue.isEmpty()){
            ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
            if (records.isEmpty()) {
                log.info("No More Records to Add");
                consumer.close();
            }
            else {
                records.iterator().forEachRemaining(record -> {
                    log.info("RECORD: " + record);
                    payloadQueue.offer(record);
                });
                payload = payloadQueue.poll().value();
                log.info("Received event from KAFKA on subject {} with payload \"{}\"", subject, payload);
            }
        }
        else {
            payload = payloadQueue.poll().value();
            log.info("Received event from KAFKA on subject {} with payload \"{}\"", subject, payload);
        }

回答1:

Kafka use batch get to improve performance, it's really not necessary to set max.poll.records=1.
What you want can be easily achieved with some workaround.

solution

You can have a Queue to store the message, each time the receive button is pressed, you poll one message from the queue, if the queue is empty, you call consumer.poll to fill the queue.

code

    private Queue<ConsumerRecord<String,String>> queue=new LinkedList<>();
    KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(properties);
    public void buttonPressed(){
        if (queue.isEmpty()){
            consumer.poll(100).iterator().forEachRemaining(record->queue.offer(record));
        }else {
            System.out.println(queue.poll());
        }
    }

回答2:

You should add a Queue as suggested by @haoyu, but manually commiting consumed offsets. Otherwise an application reset may lead to data loss (because the messages have been consumed from the topic, despite not being printed into the UI).

It is recommended to read the section "Manual Offset Control" and "Storing Offsets Outside Kafka" of KafkaConsumer javadoc.

来源：https://stackoverflow.com/questions/64328346/how-to-get-messages-from-kafka-consumer-one-by-one-in-java

标签

java