Filebeat not sending updated logs to AWS elastic search

问题

I have configured filebeat(version 7.1.1) on my system to send logs to AWS Elastic Search but it is not doing following two things appropriately:

Updating it's own log file at /var/log/filebeat, and
Sending the updated logs from /var/log/*logs to AWS ES

Issue 1: When I run sudo service filebeat restart it does not update logs files. I tried running filebeat -c /etc/filebeat/filebeat.yml to update the log files. When I run this command the terminal hangs and I have to manually close the terminal and start a new session. Filebeat logs get updated after this.

Issue 2: The log files are sent only the first time and then it never gets updated. When I do curl XGET aws_elasticsearch_domain it only returns the logs sent for the first time.I checked the Kibana dashboard as well and it only has the logs from the first time.

As per my current /var/log/filebeat/filebeat log file the connection to ES was successful and the logs were send but I still cannot see the updated logs on Kibana or through curl -XGET vpc-xxxxxxx.es.amazonaws.com/_search?pretty=true&q=*:*.

filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log
  scan_frequency: 5s
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after

setup.template.settings:
  index.number_of_shards: 1

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

output.elasticsearch:
  hosts: ["vpc-xxxxxxxxxx.es.amazonaws.com:80"]
  bulk_max_size: 2
  timeout: 180

setup.kibana:
  host: ["vpc-xxxxxxxxxxx.es.amazonaws.com:80"]
  path: /_plugin/kibana

来源：https://stackoverflow.com/questions/56500069/filebeat-not-sending-updated-logs-to-aws-elastic-search

标签

ElasticSearch

filebeat

aws-elasticsearch