kafka-python: producer is not able to connect

Question

kafka-python (1.0.0) throws error while connecting to the broker. At the same time /usr/bin/kafka-console-producer and /usr/bin/kafka-console-consumer work fine.

Pyt

Answer 1

In your server.properties file make sure he Listener IP is set to your box Ip address which is accessible to remote machine. By default it is localhost

Update this line in your server.properties:

listeners=PLAINTEXT://<Your-IP-address>:9092

Also make sure you don't have a firewall which might be blocking other IP addresses to reach you. If you have sudo previleges. The try disabling the firewall.

sudo systemctl stop firewalld

Answer 2

Install kafka-python using pip install kafka-python

Steps to create kafka data pipeline:-
1. Run the Zookeeper using shell command or install zookeeperd using

sudo apt-get install zookeeperd 

This will run zookeeper as a daemon and by default listens to 2181 port

2. Run the kafka Server
3. Run the script with producer.py and consumer.py on separate consoles to see the live data.

Here are the commands to run:-

cd kafka-directory
./bin/zookeeper-server-start.sh  ./config/zookeeper.properties    
./bin/kafka-server-start.sh  ./config/server.properties

Now that you have zookeeper and kafka server running, Run the producer.py script and consumer.py

Producer.py:

from kafka import KafkaProducer import time

producer = KafkaProducer(bootstrap_servers=['localhost:9092'])
topic = 'test'
lines = ["1","2","3","4","5","6","7","8"]
for line in lines:
  try:
    producer.send(topic, bytes(line, "UTF-8")).get(timeout=10)
  except IndexError as e:
    print(e)
  continue

Consumer.py:-

from kafka import KafkaConsumer
topic = 'test'
consumer = KafkaConsumer(topic, bootstrap_servers=['localhost:9092'])
for message in consumer:
    # message value and key are raw bytes -- decode if necessary!
    # e.g., for unicode: `message.value.decode('utf-8')`
    # print ("%s:%d:%d: key=%s value=%s" % (message.topic, message.partition,
    #                                       message.offset, message.key,
    #                                       message.value))
    print(message)

Now run the producer.py and consumer.py in separate terminals to see the live data..!

Note: Above producer.py script runs once only to run it forever, use while loop and use time module.

Answer 3

A host could have multiple dns aliases. Any of them would work for ssh or ping test. However kafka connection should use the alias that matches advertised.host.name in server.properties file of the broker.

I was using a different alias in bootstrap_servers parameter. Hence an error. Once I changed the call to use advertised.hostname, the problem was solved

Answer 4

I had the same problem.

I solved the problem with hint of user3503929.

The kafka server was installed on windows.

server.properties

...
host.name = 0.0.0.0
...

.

producer = KafkaProducer(bootstrap_servers='192.168.1.3:9092',         
                                         value_serializer=str.encode)
producer.send('test', value='aaa')
producer.close()
print("DONE.")

There was no problem with the processing in the windows kafka client. However, when I send a message to topic using kafka-python in ubuntu, a NoBrokersAvailable exception is raised.

Add the following settings to server.properties.

...
advertised.host.name = 192.168.1.3
...

It runs successfully in the same code. I spent three hours because of this.

Thanks

Answer 5

I had a similar problem. In my case, broker hostname was unresolvable on the client side . Try to explicitly set advertised.host.name in the configuration file.

Answer 6

I had a similar problem and removing the port from the bootstrap_servers helped.

consumer = KafkaConsumer('my_topic',
                     #group_id='x',
                     bootstrap_servers='kafka.com')