Does Kafka support request response messaging

Question

I am investigating Kafka 9 as a hobby project and completed a few \"Hello World\" type examples.

I have got to thinking about Real World Kafka applications based on

Answer 1

Easier! You can only write on zookeeper that the UUID X should be answered on partition Y, and make the producer that sent that UUID consume the partition Y... Does that make sense?

Answer 2

I never tried that, but in theory, if you before starting any produce, produce some messages keyed with numbers from 0 to number of partitions from answer topic and your producers are already consumers of that topic, so every producer would receive at least one of those messages. So you could store that key on each producer and publish it with the uuid... After the process on the consumer, it can publish the answer (on the answer topic) with the uuid and keyed with that same key sent with it, so it will be got by the same producer that sent it... Once all messages with the same key is published in the same partition...

Answer 3

I think you need a well defined shard key of the service that invokes the request. Your request should contain this shard key and the name of the topic where to post response. Also you should create some sort of state machine and when a message regarding your task comes you transition to some state... this would be for strict async design

Answer 4

Even though Kafka provides convenience methods to persist the committed offsets for a given consumer group, you're not required to use that behavior and can write your own if you feel the need. Even so, the use of Kafka the way you've described it is a bit awkward for the use case as each client needs to repeatedly search the topic for a specific response. That's inefficient at best.

You could break the problem into two parts, continuing to use Kafka to deliver requests to and responses from your server. The only piece you'd need to add would be some sort of API layer that your clients talk to and which encapsulates the Kafka-specific logic from your clients. This layer would need a local DB (relational or NoSQL) that could store responses by uuid making it very fast and easy for the API to answer whether a response is available for a specific uuid.