How do I view the consumed messages of Kafka in Nifi?

Question

I have started a Nifi process(Consume Kafka) and connected it to a topic. It is running but I am not able to (don\'t know) where can I view the messages?

Answer 1

ConsumeKafka processor runs and generates flowfile for each message. Only when you connect a processor to other components like another processor or an output port, will you be able to visualize the data being moved through.

For starters you can try this:

• Connect ConsumeKafka with LogAttribute or any other processor for that matter.
• Stop or disable the LogAttribute processor.
• Now when you start ConsumeKafka, all the received messages from the configured Kafka topic will be queued up in the form of flowfiles.
• Right click that relationship where the flowfiles are queued up and click List Queue and you can access the queue.
• Click any item on the queue, a context menu will come up. Click View button and you can see the data.

This whole explanation of "viewing" the Kafka message is just to help you in debugging and get started with NiFi. Ideally you would be using other NiFi processors to work out your usecase.

Example

You receive messages from Kafka and wants to write it to MongoDB, so you can have the flow as: Note: There are record based processors like ConsumeKafkaRecord and PutMongoRecord but they are basically doing the same thing with more enhancements. Since you're new to this, I have suggested a simple flow. You can find details about the Record based processors here and try that.

Answer 2

You might need to consume messages --from-beginning if those messages have been consumed before (and therefore offsets have been committed).

On GetKafka processor, there is a property Auto Offset Reset which should be set to smallest which is the equivalent of --from-beginning in Kafka Console Consumer.