kafka集群搭建kafka_2.11-0.11.0.0.tgz
此时zookeeper集群已经搭建好并启动状态,在3台虚拟机上
(1)配置文件:
config/server.properties:此时要主要原来文件本身已经带有这些了,要先把以前的注释掉,或者直接在原有基础上改(遇到的坑)
broker.id=1
listeners=PLAINTEXT://192.168.132.130:9092(这里是自己主机ip地址)
log.dirs=/usr/kafka/dataLog
zookeeper.connect=192.168.132.130:2181,192.168.132.131:2181,192.168.132.132:2181
producer.properties:
bootstrap.servers=192.168.132.130:9092,192.168.132.131:9092,192.168.132.132:9092
consumer.properties:
zookeeper.connect=192.168.132.130:2181,192.168.132.131:2181,192.168.132.132:2181
(2)配置成功后启动,每个都启动一下
nohup bin/kafka-server-start.sh config/server.properties &
jps命令查看是否启动成功:
8115 Kafka
8013 QuorumPeerMain
8829 Jps
创建topic命令:
kafka-topics --create --zookeeper-server hadoop1:9092,hadoop2:9092,hadoop3:9092 --replication-factor 1 --partitions 1 --topic topicname
replication-factor: 创建的副本数,这个使用来备份的.副本数不能大于broker数
partitions 1: 创建的分区数.根据实际情况创建
查看topic列表命令:
bin/kafka-topics.sh --zookeeper 192.168.132.130:2181,192.168.132.131:2181,192.168.132.132:2181 --list
(3)错误类型
WARN Connection to node -3 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
解决:
将端口号改一下,原来2、3虚拟机上 的消费者接受消息的命令端口是9092,此时消费者命令会出现No route to host和Connection refused错误,改完端口就好了能正常接受消息
生产者:
bin/kafka-console-producer.sh --broker-list hadoop1:9092,hadoop2:9092,hadoop3:9092 --topic test
消费者:
bin/kafka-console-consumer.sh --zookeeper hadoop1:2181,hadoop2:2181,hadoop3:2181 --topic test --from-beginning
目前这样就能实现从生产者输入的信息,能在消费者那看到
查看每个节点的信息
bin/kafka-topics.sh --describe --zookeeper 192.168.132.130:2181 --topic test
得到:
Topic: test PartitionCount:3 ReplicationFactor:3 Configs:
Topic: test Partition: 0 Leader: 3 Replicas: 3,1,2 Isr: 2,3
Topic: test Partition: 1 Leader: 2 Replicas: 1,2,3 Isr: 2,3
Topic: test Partition: 2 Leader: 2 Replicas: 2,3,1 Isr: 2,3
来源:CSDN
作者:j0j1j
链接:https://blog.csdn.net/rjuan/article/details/104040415