Mutliple config files causing duplicate message

问题

I have a Logstash machine running in AWS. In Logstash I have 3 config files each having 1 input defined on them. These inputs are reading logs from following sources

From s3
From http input
From filebeat

The problem is that I am getting duplicate messages in Kibana. So for 1 message generated by Filebeat I am seeing 3 messages in Kibana. I tried to remove 1 config file and the count got reduced to 2. So I am pretty sure that this is due to these config files.

What is confusing me is that why this is happening. I have separate input defined on all 3 config files, still getting duplicate messages. These are the input section of all 3 config files.

s3 input

input {
  s3 {
     bucket => "elb-logs"
     region => "us-east-1"
     prefix => "demo/AWSLogs/792177735214/"
     type   => "elb-logs"
     delete => true
  }

}

Http input

input {
  http {
     type   => "frontend-logs"
     codec  => "json"
  }
}

Filebeat

input {
  beats {
    port => "5043"
  }
}

For all 3 config files there is common output section i.e.

output {
    elasticsearch { hosts => [ "10.0.0.1:9200" ] }
}

回答1:

Logstash will concatenate the three config files together (s3 input, Http input, Filebeat) and see three output sections.

The three output sections are not related to the specific inputs - instead Logstash will send an input from any one of the three sources to all of the configured outputs. As a result your message will be output three times to the same destination.

I would create a separate, single output config file and remove the output section from your 3 input config files.

来源：https://stackoverflow.com/questions/43781955/mutliple-config-files-causing-duplicate-message

标签

ElasticSearch

logstash

kibana

filebeat