04 友盟项目--access.log日志通过flume传入kafka集群

收集access.log数据到kafka集群

如果必要

清除原来的日志信息

xcall.sh "echo > /usr/local/openresty/nginx/logs/access.log"

开zk

开redis

开kafka

1 .kafka 主题

创建主题

kafka-topics.sh --zookeeper s102:2181 --partitions 4 --replication-factor 3 --create --topic big12-umeng-raw-logs

查看主题

kafka-topics.sh --zookeeper s102:2181 --list

创建kafka消费者 ----此步可以省略

kafka-console-consumer.sh --zookeeper s102:2181 --topic big12-umeng-raw-logs

2 配置flume，收集access.log文件到kafka集群

/soft/flume/conf sudo nano umeng_nginx_to_kafka.conf

flume配置文件

a1.sources = r1
a1.channels = c1
a1.sinks = k1
 
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /usr/local/openresty/nginx/logs/access.log
 
a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000
 
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.kafka.topic = big12-umeng-raw-logs
a1.sinks.k1.kafka.bootstrap.servers = s102:9092
a1.sinks.k1.kafka.flumeBatchSize = 20
#acks 0:不需要回执 1:leader写入磁盘后回执 -1:所有节点写入磁盘后回执
a1.sinks.k1.kafka.producer.acks = 1
a1.sinks.k1.kafka.producer.linger.ms = 0
 
a1.sources.r1.channels=c1
a1.sinks.k1.channel=c1

发送配置文件到s102