Kafka access inside and outside docker

Question

I\'m trying to start a kafka service using docker-compose, and it should be able to be accessed inside and outside docker. So, it should be matter of setting the right adver

Answer 1

What I needed to do was to declare the LISTENERS as both binding to the docker host, and then advertise them differently - one to the docker network, one to the host:

services:
  zookeeper:
    image: confluentinc/cp-zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_SYNC_LIMIT: 2
  kafka:
    image: confluentinc/cp-kafka
    ports:
      - 9094:9094
    depends_on:
      - zookeeper
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: INTERNAL://kafka:9092,OUTSIDE://kafka:9094
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,OUTSIDE://localhost:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL

Answer 2

Agreed to the answer from @Robin Moffatt that outside port 9094 is mapped to container's 9092.

In addition to removing that configuration from docker-compose.yml, the outside port 9094 should be mapped to container's 9094.

ports:
  - "9094:9094"

PS: I couldn't add this as a comment to previous answer, hence adding it as a new answer.

Answer 3

Your config of the listeners looks correct, the problem is with your Docker Compose:

ports:
  - "9094:9092"

You’re mapping 9094 (‘Outside’) back to 9092 (‘Inside’) and thus when you connect you’re connecting to the ‘Inside’ listener. If you remove this line of config then your listener set up should work as intended.

For more info, see http://rmoff.net/2018/08/02/kafka-listeners-explained/