Why I cannot connect to Kafka from outside?

Question

I am running kafka on ec2 instance. So amazon ec2 instance has two ips one is internal ip and second one is for external use.

I created producer from local machine,

Answer 1

The easiest way how to reach your Kafka server (version kafka_2.11-1.0.0) on EC2 from consumer in external network is to change the properties file

kafka_2.11-1.0.0/config/server.properties

And modify the following line

listeners=PLAINTEXT://ec2-XXX-XXX-XXX-XXX.eu-central-1.compute.amazonaws.com:9092

Using your public address

Verified on 2.11-2.0.0

Answer 2

Below are the steps to connect Kafka from outside of EC2 instance.

1. Open Kafka server properties file on EC2.

   /kafka_2.11-2.0.0/config/server.properties

2. Set the value of advertised.listeners to

   advertised.listeners=PLAINTEXT://ec2-xx-xxx-xxx-xx.compute-1.amazonaws.com:9092

   This should be your Public DNS (IPv4) of EC2 instance.

3. Stop Kafka server.

4. Start Kafka server to see above configuration changes in action.

5. Now you can connect to your Kafka of EC2 instance from outside or from your localhost.

   Tried and tested on kafka_2.11-2.0.0

Answer 3

SSH to your EC2 instance or wheverver you're hosting Kafka.

sudo nano /etc/hosts

Add:

127.0.0.1 <your-host-name> localhost

In my case it's:

127.0.0.1 ec2-12-34-56-78.ap-southeast-1.compute.amazonaws.com

Save and exit.

Answer 4

If you want to access from LAN, change following 2 files-

1. In config/server.properties:

   advertised.listeners=PLAINTEXT://server.ip.in.lan:9092
   

2. In config/producer.properties:

   bootstrap.servers=server.ip.in.lan:9092
   

In my case, the server.ip.in.lan value was 192.168.15.150

Answer 5

I solved this problem, by setting advertised.host.name in server.properties and metadata.broker.list in producer.properties to public IP address and host.name to 0.0.0.0.

Answer 6

In the Kafka FAQ (updated for new properties) you can read:

When a broker starts up, it registers its ip/port in ZK. You need to make sure the registered ip is consistent with what's listed in bootstrap.servers in the producer config. By default, the registered ip is given by InetAddress.getLocalHost.getHostAddress(). Typically, this should return the real ip of the host. However, sometimes (e.g., in EC2), the returned ip is an internal one and can't be connected to from outside. The solution is to explicitly set the host ip and port to be registered in ZK by setting the advertised.listeners property in server.properties.