Connect to Kafka running in Docker

Question

I setup a single node Kafka Docker container on my local machine like it is described in the Confluent documentation (steps 2-3).

In addition, I also exposed Zookeep

Answer 1

When you first connect to a kafka node, it will give you back all the kafka node and the url where to connect. Then your application will try to connect to every kafka directly.

Issue is always what is the kafka will give you as url ? It's why there is the KAFKA_ADVERTISED_LISTENERS which will be used by kafka to tell the world how it can be accessed.

Now for your use-case, there is multiple small stuff to think about:

Let say you set plaintext://kafka:9092

• This is OK if you have an application in your docker compose that use kafka. This application will get from kafka the URL with kafka that is resolvable through the docker network.
• If you try to connect from your main system or from another container which is not in the same docker network this will fail, as the kafka name cannot be resolved.

==> To fix this, you need to have a specific DNS server like a service discovery one, but it is big trouble for small stuff. Or you set manually the kafka name to the container ip in each /etc/hosts

If you set plaintext://localhost:9092

• This will be ok on your system if you have a port mapping ( -p 9092:9092 when launching kafka)
• This will fail if you test from an application on a container (same docker network or not) (localhost is the container itself not the kafka one)

==> If you have this and wish to use a kafka client in another container, one way to fix this is to share the network for both container (same ip)

Last option : set an IP in the name: plaintext://x.y.z.a:9092

This will be ok for everybody... BUT how can you get the x.y.z.a name ?

The only way is to hardcode this ip when you launch the container: docker run .... --net confluent --ip 10.x.y.z .... Note that you need to adapt the ip to one valid ip in the confluent subnet.