Kafka in Kubernetes Cluster- How to publish/consume messages from outside of Kubernetes Cluster

Question

1. I have Kafka deployed and running in Kubernetes cluster. I am using this image from docker hub - https://hub.docker.com/r/cloudtrackinc/kubernetes-kafka/
2. I have 3

Answer 1

I had the same problem with accessing kafka from outside of k8s cluster on AWS. I manage to solve this issue by using kafka listeners feature which from version 0.10.2 supports multiple interfaces.

here is how I configured kafka container.

    ports:
    - containerPort: 9092
    - containerPort: 9093
    env:
    - name: KAFKA_ZOOKEEPER_CONNECT
      value: "zookeeper:2181"
    - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
      value: "INTERNAL_PLAINTEXT:PLAINTEXT,EXTERNAL_PLAINTEXT:PLAINTEXT"
    - name: KAFKA_ADVERTISED_LISTENERS
      value: "INTERNAL_PLAINTEXT://kafka-internal-service:9092,EXTERNAL_PLAINTEXT://123.us-east-2.elb.amazonaws.com:9093"
    - name: KAFKA_LISTENERS
      value: "INTERNAL_PLAINTEXT://0.0.0.0:9092,EXTERNAL_PLAINTEXT://0.0.0.0:9093"
    - name: KAFKA_INTER_BROKER_LISTENER_NAME
      value: "INTERNAL_PLAINTEXT"

Apart from that I configured two Services. One for internal(Headless) & one for external(LoadBalancer) communication.

Hopefully this will save people's time.