Logstash is not reading from Kafka

百般思念 提交于 2019-12-11 06:15:23

问题


I am testing a simple pipeline - Filebeat > Fafka > Logstash > File.

Logstash is not reading from Kafka, but I see Kafka has messages when i use this command -

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic MyTopic --from-beginning

My file beat configuration -

filebeat.prospectors:
- input_type: log
  paths:
    - /root/LogData/input.log

output.kafka:
  hosts: ["10.247.186.14:9092"]

  topic: MyTopic 
  partition.round_robin:
    reachable_only: false

  required_acks: 1
  compression: none
  max_message_bytes: 1000000

My Logstash configuration

input {
        kafka {
                bootstrap_servers => "localhost:9092"
                topics => ["MyTopic"]
        }

}

output {

        file {
                path => "/usr/share/logstash/test_out.log"

        }
}

My Fafka & Logstash are running on same VM, I am using Logstash docker, I started the docker of Logstash as follows -

sudo docker run -d --name logstash --expose 25826 -p 25826:25826 docker.elastic.co/logstash/logstash:5.4.0 --debug

Filebeat running on different VM.

I am creating topic as follows -

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic MyTopic

来源:https://stackoverflow.com/questions/44489847/logstash-is-not-reading-from-kafka

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!