消息队列学习03：Zookeeper、Kafka在Linux平台的安装和配置

安装Zookeeper

下载安装包

　　在Apache Zookeeper镜像站点下载安装包；我下载的是3.4.14版本

安装Zookeeper

　　1-通过XFTP将zookeeper-3.4.14.tar安装包上传到Linux服务器usr/local目录下

　　2-到usr/local目录下，确认是否成功

[root@izbp1aum9uyt0x56d3vepwz ~]# cd /usr/local/
[root@izbp1aum9uyt0x56d3vepwz local]# ls -al
total 36860
drwxr-xr-x. 15 root root     4096 Oct 16 19:13 .
drwxr-xr-x. 13 root root     4096 Mar 13  2019 ..
drwxr-xr-x   7 root root     4096 Aug 31 03:59 aegis
drwxr-xr-x.  2 root root     4096 Apr 12  2019 bin
drwxr-xr-x   4 root root     4096 Aug  7 09:15 cloudmonitor
drwxr-xr-x.  2 root root     4096 Apr 11  2018 etc
drwxr-xr-x.  2 root root     4096 Apr 11  2018 games
drwxr-xr-x.  2 root root     4096 Apr 11  2018 include
drwxr-xr-x.  2 root root     4096 Apr 11  2018 lib
drwxr-xr-x.  2 root root     4096 Apr 11  2018 lib64
drwxr-xr-x.  2 root root     4096 Apr 11  2018 libexec
drwxrwxr-x   6 root root     4096 Aug  7 14:24 redis
drwxr-xr-x.  2 root root     4096 Apr 11  2018 sbin
drwxr-xr-x.  6 root root     4096 Apr 11  2018 share
drwxr-xr-x.  2 root root     4096 Apr 11  2018 src
-rw-r--r--   1 root root 37676320 Oct 16 19:13 zookeeper-3.4.14.tar.gz

　　3-将zookeeper-3.4.14.tar.gz包解压至/usr/local/路径

[root@izbp1aum9uyt0x56d3vepwz local]# tar -zxvf zookeeper-3.4.14.tar.gz 
……………………省略…………………………
zookeeper-3.4.14/README_packaging.txt
zookeeper-3.4.14/build.xml
zookeeper-3.4.14/ivy.xml
zookeeper-3.4.14/ivysettings.xml
zookeeper-3.4.14/pom.xml
zookeeper-3.4.14/zookeeper-3.4.14.jar
zookeeper-3.4.14/zookeeper-3.4.14.jar.md5
zookeeper-3.4.14/zookeeper-3.4.14.jar.sha1
zookeeper-3.4.14/zookeeper-3.4.14.jar.asc

　　4-进入/usr/local/zookeeper-3.4.14/conf目录，复制文件zoo_sample.cfg并命名为zoo.cfg

[root@izbp1aum9uyt0x56d3vepwz zookeeper-3.4.14]# cd /usr/local/zookeeper-3.4.14/conf/
[root@izbp1aum9uyt0x56d3vepwz conf]# cp zoo_sample.cfg  zoo.cfg

　　5-在zookeeper-3.4.14文件夹下创建data、logs两个目录

[root@izbp1aum9uyt0x56d3vepwz conf]# cd /usr/local/zookeeper-3.4.14
[root@izbp1aum9uyt0x56d3vepwz zookeeper-3.4.14]# mkdir data
[root@izbp1aum9uyt0x56d3vepwz zookeeper-3.4.14]# mkdir logs

　　6-配置zoo.cfg文件：dataDir=/usr/local/zookeeper-3.4.14/data 和 dataLogDir=/usr/local/zookeeper-3.4.14/logs；这里我是通过XFTP下载zoo.cfg到本地，修改后上传覆盖源文件；也可以选择使用Linux vi指令

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/usr/local/zookeeper-3.4.14/data
dataLogDir=/usr/local/zookeeper-3.4.14/logs
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

　　7-在/etc/profile尾部追加如下内容，即配置Linux下zookeeper环境变量

# zookeeper-3.4.14 config start
export ZOOKEEPER_HOME=/usr/local/zookeeper-3.4.14
export PATH=$ZOOKEEPER_HOME/bin:$PATH
# zookeeper-3.4.14 config end

　　8-使/etc/profile文件立即生效

[root@ etc]# source /etc/profile

　　9-启动Zookepper：输入zkServer.sh start，启动Zookeeper服务，如打印如下信息则表示启动成功

[root@izbp1aum9uyt0x56d3vepwz zookeeper-3.4.14]# zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper-3.4.14/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

常用的指令

　　1-重启Zookeeper：zkServer.sh restart

[root@izbp1aum9uyt0x56d3vepwz zookeeper-3.4.14]# zkServer.sh restart
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper-3.4.14/bin/../conf/zoo.cfg
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper-3.4.14/bin/../conf/zoo.cfg
Stopping zookeeper ... no zookeeper to stop (could not find file /tmp/zookeeper/zookeeper_server.pid)
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper-3.4.14/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

　　2-停止Zookeeper：zkServer.sh stop

Using config: /usr/local/zookeeper-3.4.14/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED

　　3-查看Zookeeper状态：zkServer.sh status

[root@izbp1aum9uyt0x56d3vepwz zookeeper-3.4.14]# zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper-3.4.14/bin/../conf/zoo.cfg
Mode: standalone

安装Kafka

下载安装包

　　1-进入Apache Kafka官方网站，此处选择Kafka 2.0.0版本，对应的Scala版本是2.12，找到kafka_2.12-2.0.0.tgz，点击下载

安装Kafka

　　1-通过XFTP将kafka_2.12-2.0.0安装包上传到Linux服务器usr/local目录下

　　2-将kafka_2.12-2.0.0.tgz包解压至/usr/local/路径

[root@izbp1aum9uyt0x56d3vepwz local]# cd /usr/local
[root@izbp1aum9uyt0x56d3vepwz local]# tar -zxvf kafka_2.12-2.0.0.tgz
…………省略………………
kafka_2.12-2.0.0/libs/connect-basic-auth-extension-2.0.0.jar
kafka_2.12-2.0.0/libs/kafka-streams-2.0.0.jar
kafka_2.12-2.0.0/libs/rocksdbjni-5.7.3.jar
kafka_2.12-2.0.0/libs/kafka-streams-scala_2.12-2.0.0.jar
kafka_2.12-2.0.0/libs/kafka-streams-test-utils-2.0.0.jar
kafka_2.12-2.0.0/libs/kafka-streams-examples-2.0.0.jar

　　3-创建日志目录/kafka_2.12-2.0.0/logs

[root@izbp1aum9uyt0x56d3vepwz local]# cd /usr/local/kafka_2.12-2.0.0
[root@izbp1aum9uyt0x56d3vepwz kafka_2.12-2.0.0]# mkdir logs

　　4-修改/kafka_2.12-2.0.0/config/server.properties文件,分别修改三个地方: listeners、advertised.listeners、log.dirs；其中外网IP的设置是为了可以远程使用当前服务器的kafka(当前需要先开放对应的zookeeper和kafka端口)

#内网IP
listeners=PLAINTEXT://192.168.126.XXX:9092

#外网IP
advertised.listeners=PLAINTEXT://47.111.XXX.6:9092

#日志记录
log.dirs=/usr/local/kafka_2.12-2.0.0/logs

启动Kafka

　　1-在当前目录下输入 ./bin/kafka-server-start.sh ./config/server.properties，启动Kafka，可以启动，但是一旦向继续输入指令，则会强制退出kafka

[root@izbp1aum9uyt0x56d3vepwz kafka_2.12-2.0.0]# ./bin/kafka-server-start.sh ./config/server.properties
…………………省略………………………………
[2019-10-16 20:03:07,873] INFO [GroupMetadataManager brokerId=0] Removed 0 expired offsets in 60 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2019-10-16 20:03:07,942] INFO [TransactionCoordinator id=0] Starting up. (kafka.coordinator.transaction.TransactionCoordinator)
[2019-10-16 20:03:07,952] INFO [TransactionCoordinator id=0] Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)
[2019-10-16 20:03:07,974] INFO [Transaction Marker Channel Manager 0]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
[2019-10-16 20:03:08,062] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
[2019-10-16 20:03:08,097] INFO [SocketServer brokerId=0] Started processors for 1 acceptors (kafka.network.SocketServer)
[2019-10-16 20:03:08,102] INFO Kafka version : 2.0.0 (org.apache.kafka.common.utils.AppInfoParser)
[2019-10-16 20:03:08,102] INFO Kafka commitId : 3402a8361b734732 (org.apache.kafka.common.utils.AppInfoParser)
[2019-10-16 20:03:08,104] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

…………………省略……………………………
[2019-10-16 20:01:29,339] INFO [ThrottledChannelReaper-Request]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2019-10-16 20:01:30,332] INFO [ThrottledChannelReaper-Request]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2019-10-16 20:01:30,332] INFO [ThrottledChannelReaper-Request]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2019-10-16 20:01:30,334] INFO [SocketServer brokerId=0] Shutting down socket server (kafka.network.SocketServer)
[2019-10-16 20:01:30,377] INFO [SocketServer brokerId=0] Shutdown completed (kafka.network.SocketServer)
[2019-10-16 20:01:30,384] INFO [KafkaServer id=0] shut down completed (kafka.server.KafkaServer)

　　2-使用nohup &操作，即 nohup ./bin/kafka-server-start.sh ./config/server.properties & 如下(nohup表示不挂断地运行命令， &表示在后台运行)

[root@izbp1aum9uyt0x56d3vepwz kafka_2.12-2.0.0]# nohup ./bin/kafka-server-start.sh  ./config/server.properties &
[1] 16704
[root@izbp1aum9uyt0x56d3vepwz kafka_2.12-2.0.0]# nohup: ignoring input and appending output to ‘nohup.out’
^C
[root@izbp1aum9uyt0x56d3vepwz kafka_2.12-2.0.0]#

　　3-创建topic：./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test　

[root@izbp1aum9uyt0x56d3vepwz kafka_2.12-2.0.0]# cd /usr/local/kafka_2.12-2.0.0/bin/
[root@izbp1aum9uyt0x56d3vepwz bin]# ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Created topic "test".
[root@izbp1aum9uyt0x56d3vepwz bin]# ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic tangyj
Created topic "tangyj".

　　4-查看topic：./kafka-topics.sh --list --zookeeper localhost:2181

[root@izbp1aum9uyt0x56d3vepwz bin]# ./kafka-topics.sh --list --zookeeper localhost:2181
tangyj
test

　　5-启动producer(使用topic: test)：并发送消息hello tyj 启动指令：./kafka-console-producer.sh --broker-list 192.168.126.XXX:9092 --topic test

[root@izbp1aum9uyt0x56d3vepwz bin]#  ./kafka-console-producer.sh --broker-list 192.168.126.XXX:9092 --topic test
>hello tyj  #发送消息

　　6-启动consumer(使用topic: test)，接收消息；启动指令： ./kafka-console-consumer.sh --bootstrap-server 192.168.126.XXX:9092 --topic test --from-beginning

[root@izbp1aum9uyt0x56d3vepwz bin]#  ./kafka-console-consumer.sh --bootstrap-server 192.168.126.XXX:9092 --topic test --from-beginning
hello tyj #接收消息