How i can config multiline in logstash 5.1.2 for tomcat/java

问题

I use a 5.1.2 verisón of logstash, filebeat, elasticsearch... "ELK"

I try send logs from tomcat server (catalina.out and apps-java logs) but can´t because have problems of config of logstash multiline filter/codec.

I follow this instructions
https://blog.lanyonm.org/articles/2014/01/12/logstash-multiline-tomcat-log-parsing.html

Logstash.conf is this:

input {
    beats {
    port => 9000
    }
}

filter {
  if [type] == "tomcat-pro" {
    codec => "multiline" {
      patterns_dir => "/opt/logstash/patterns"
      pattern => "(^%{TOMCAT_DATESTAMP})|(^%{CATALINA_DATESTAMP})"
      negate => true
      what => "previous"
    }
  }
}

output {
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "tomcat-pro"
    }   
}

Logstash receives files of filebeat.
Filebeat.yml

filebeat.prospectors:
- input_type: log
  document_type: tomcat-pro
  paths:
  - /opt/tomcat-test/logs/catalina.out

When i start the service the console show me this:

[2017-01-26T13:10:33,712][ERROR][logstash.agent           ] fetched an invalid config {:config=>"input {\n    beats {\n    port => 9000\n    }\n}\n\nfilter {\n  if [type] == \"tomcat-pro\" {\n    codec => \"multiline\" {\n      patterns_dir => \"/opt/logstash/patterns\"\n      pattern => \"(^%{TOMCAT_DATESTAMP})|(^%{CATALINA_DATESTAMP})\"\n      negate => true\n      what => \"previous\"\n    }\n  }\n}\n\noutput {\n    elasticsearch {\n        hosts => [\"localhost:9200\"]\n        index => \"tomcat-pro\"\n    }   \n}\n", :reason=>"Expected one of #, { at line 9, column 11 (byte 96) after filter {\n  if [type] == \"tomcat-pro\" {\n    codec "}

Summary:

fetched an invalid config
reason=>"Expected one of #, { at line 9, column 11 (byte 96) after filter {\n  if [type] == \"tomcat-pro\" {\n    codec "}

I read in google that is recommended to use multiline in filebeat rather than in logstash, but i dont config very well...

Someone can help me? :(

PD: Im spanish, sorry for "google translate". Si puedes responder en español, sería mucho mejor ;)

回答1:

I think doing the multiline processing in Filebeat is the way to go, so instead of debugging the Logstash configuration error you posted I will show a Filebeat configuration where Filebeat combines the lines before shipping the event.

If you were only using Logstash for the multiline filter, then you could just output directly to Elasticsearch from Filebeat. But if you do need to output to Logstash please follow the instructions for configuring Filebeat to be used with Logstash.

The pattern I use below hasn't been thoroughly tested so please test it against the actual logs.

filebeat.prospectors:
- document_type: catalina-wine-mixer
  paths:
  - /opt/tomcat-test/logs/catalina.out
  multiline.pattern: '^([0-9]{4}-[0-9]{2}-[0-9]{2})|([J|F|M|A|M|S|O|N|D][a-z]{2} [0-9]{1,2}, [0-9]{2})'
  multiline.negate: true
  multiline.match: after

output.elasticsearch:
  hosts: ['http://localhost:9200']

回答2:

The answered pattern didn't quite work for me, for catalina/tomcat logs, i'm currently using the following filebeat.yml pattern:

multiline.pattern: '^[[:alpha:]]{3} [0-9]{2}, [0-9]{4}'
multiline.negate: true
multiline.match: after

来源：https://stackoverflow.com/questions/41873228/how-i-can-config-multiline-in-logstash-5-1-2-for-tomcat-java

标签

Tomcat

logstash

elastic-stack

filebeat