kafka集群

时光总嘲笑我的痴心妄想 提交于 2020-01-20 04:41:36

kafka集群搭建kafka_2.11-0.11.0.0.tgz

此时zookeeper集群已经搭建好并启动状态,在3台虚拟机上

(1)配置文件:

config/server.properties:此时要主要原来文件本身已经带有这些了,要先把以前的注释掉,或者直接在原有基础上改(遇到的坑)

broker.id=1
listeners=PLAINTEXT://192.168.132.130:9092(这里是自己主机ip地址)
log.dirs=/usr/kafka/dataLog
zookeeper.connect=192.168.132.130:2181,192.168.132.131:2181,192.168.132.132:2181

producer.properties:

bootstrap.servers=192.168.132.130:9092,192.168.132.131:9092,192.168.132.132:9092

consumer.properties:

zookeeper.connect=192.168.132.130:2181,192.168.132.131:2181,192.168.132.132:2181

(2)配置成功后启动,每个都启动一下

nohup bin/kafka-server-start.sh config/server.properties &

jps命令查看是否启动成功:

8115 Kafka
8013 QuorumPeerMain
8829 Jps

创建topic命令:

kafka-topics --create --zookeeper-server hadoop1:9092,hadoop2:9092,hadoop3:9092 --replication-factor 1 --partitions 1 --topic topicname

replication-factor: 创建的副本数,这个使用来备份的.副本数不能大于broker数

partitions 1: 创建的分区数.根据实际情况创建

查看topic列表命令:

bin/kafka-topics.sh --zookeeper 192.168.132.130:2181,192.168.132.131:2181,192.168.132.132:2181 --list

(3)错误类型

WARN Connection to node -3 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

解决:

将端口号改一下,原来2、3虚拟机上 的消费者接受消息的命令端口是9092,此时消费者命令会出现No route to host和Connection refused错误,改完端口就好了能正常接受消息

生产者:

bin/kafka-console-producer.sh --broker-list hadoop1:9092,hadoop2:9092,hadoop3:9092 --topic test

消费者:

bin/kafka-console-consumer.sh --zookeeper hadoop1:2181,hadoop2:2181,hadoop3:2181 --topic test --from-beginning

目前这样就能实现从生产者输入的信息,能在消费者那看到

查看每个节点的信息

bin/kafka-topics.sh --describe --zookeeper 192.168.132.130:2181 --topic test

得到:

Topic: test	PartitionCount:3	ReplicationFactor:3	Configs:
Topic: test	Partition: 0	Leader: 3	Replicas: 3,1,2	Isr: 2,3
Topic:  test	Partition: 1	Leader: 2	Replicas: 1,2,3	Isr: 2,3
Topic:  test	Partition: 2	Leader: 2	Replicas: 2,3,1	Isr: 2,3
易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!