Error creating Kafka Topics- Replication factor larger than available brokers

Question

I am trying to create topics in Kafka by following the guide on Apache Kafka website through command line. While running the command:

bin/kafka-topics.sh --c         


        

Answer 1

I had the same problem but finally I tried the following steps which worked for me:

1)Create the chroot in the Zookeeper with the following commands:

zkCli.sh -server localhost:2181

create /kafka1 []

2)For the kafka brokers config , modify zookeeper.connect and add chroot path : For eg.

In server.properties :

zookeeper.connect=localhost:2181/kafka1

3)Start Zookeeper

4)Start Kafka Server

5) Create a topic as below: bin/kafka-topic.sh --create -zookeeper localhost:2181/kafka1 --replication-factor 1 --partitions 1 --topic data

6)Confirm the created topic: bin/kafka-topics.sh --list --zookeeper localhost:2181/kafka1

I followed below for reference: https://community.hortonworks.com/questions/102132/run-multiple-broker-versions-in-the-same-cluster.html

Answer 2

In my case the error was caused by the fact that I started creating topics before the zookeeper became available. Since I did everything in a script, introducing some latency before creating topics helped. If your zookeeper client port is 2181, you can check the availability of your brokers by:

echo dump | nc zookeeper 2181 | grep brokers

Answer 3

Your Zookeeper is unable to see any Kafka brokers.Try logging in to your Zookeepr console : bin/zkCli.sh -server localhost:2181 Run [zk: localhost:2181(CONNECTED) 2] ls /brokers/ids on the console. It must be not be showing up any brokers. You need to restart all your Kafka brokers, and check for the kafka process log. Sometimes, even when the process shows as running (ps -ef | grep kafka), it may not have really started completely, because of which Zookeeper is unable to see it. I had the same issue with Apache Kafka distribution 0.11.0.

Answer 4

• Check if all the brokers are registered to the Zookeeper:

  zookeeper-shell.sh localhost:2181 ls /brokers/ids

• Most probably some ID's are missing in the brackets []

• Login to the machine where this ID resides and restart the broker:

  kafka-server-stop.sh
  nohup kafka-server-start.sh ../config/server.properties > /dev/null 2>&1 &
  

• Check again if the Zookeeper can now see all your brokers properly.
• Retry your create topic command.

Hope it helps.

Answer 5

Seems like your server not started , check logs for issue

Kill process

sudo fuser -k 2181/tcp

run zookeeper

bin/zookeeper-server-start.sh config/zookeeper.properties

Run Kafka

bin/kafka-server-start.sh config/server.properties  ' 

Answer 6

make sure the kafka server has started properly. If you are using -dameon parameter to start kafka server as daemon. Try to remove it and see if there are any errors during the startup.

The issue I ran into turned out to be a file access issue, where the user runs kafka doesn't have access to the log directory I configured. After I granted the access, it just works!