NoBrokersAvailable: NoBrokersAvailable-Kafka Error

Question

i have already started to learn Kafka. Trying basic operations on it. I have stucked on a point which about the 'Brokers'.

My kafka is running but when i want to create a partition.

 from kafka import TopicPartition
(ERROR THERE) consumer = KafkaConsumer(bootstrap_servers='localhost:1234')
 consumer.assign([TopicPartition('foobar', 2)])
 msg = next(consumer)

traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python2.7/dist-packages/kafka/consumer/group.py", line 284, in init self._client = KafkaClient(metrics=self._metrics, **self.config) File "/usr/local/lib/python2.7/dist-packages/kafka/client_async.py", line 202, in init self.config['api_version'] = self.check_version(timeout=check_timeout) File "/usr/local/lib/python2.7/dist-packages/kafka/client_async.py", line 791, in check_version raise Errors.NoBrokersAvailable() kafka.errors.NoBrokersAvailable: NoBrokersAvailable

Answer 1

You cannot create partitions within a consumer. Partitions are created when you create a topic. For example, using command line tool:

bin/kafka-topics.sh \
  --zookeeper localhost:2181 \
  --create --topic myNewTopic \
  --partitions 10 \
  --replication-factor 3

This creates a new topic "myNewTopic" with 10 partitions (numbered from 0 to 9) and replication factor 3. (see http://docs.confluent.io/3.0.0/kafka/post-deployment.html#admin-operations and https://kafka.apache.org/documentation.html#quickstart_createtopic)

Within your consumer, if you call assign(), it means you want to consume the corresponding partition and this partition must exist already.

Answer 2

It looks like you want to start consuming messages instead of creating partions. Nevertheless - can you reach kafka at port 1234? 9092 is kafkas default port maybe you can try this one. If you found the right port but your application still produces errors you can try to use a console consumer to test your setup:

bin/kafka-console-producer.sh --broker-list localhost:<yourportnumber> --topic foobar

The console consumer is part of the standard kafka distribution. Maybe that gets you a little closer to the source of the problem.

Answer 3

I also had same error during kafka streaming. Below example resolved my error. We need to define API version in KafkaProducer.

KafkaProducer(bootstrap_servers=['localhost:9092'],
api_version=(0,11,5),
value_serializer=lambda x:
dumps(x).encode('utf-8'))

Answer 4

Don't know if this answer is still relevant but recently resolved this same problem in a VBox VM broker not reachable from host Windows OS. Since you have mentioned bootsrap_servers in KafkaConsumer, I assume you are using at least kafka 0.10.0.0

Please look for the advertised.listeners property in server.properties file and set it to PLAINTEXT://localhost:9092 or PLAINTEXT://<broker_ip>:9092

But before you set that make sure your broker is reachable from the environment where your consumer is running (by doing ping localhost).

Also, you need to restart the kafka-server and consumer/producer (whatever is running) and try sending/receiving.

For example, if you are running VM, you may like to use Host-only adapter to make the broker reachable from host machine

NOTE: This configuration works for Kafka Server >= 0.10.X.X but not for 0.8.2.X. Haven't checked for 0.9.0.X