Logstash date invalid format

我只是一个虾纸丫 提交于 2019-12-11 14:50:12

问题


Trying to parse logs from a rsylog server and insert them to elasticsearch.
My incoming logline is

Feb 13 01:17:11 xxxx xxx-xxxx_error 2016/02/13 01:17:02 [error] 13689#0: *1956118 open() "xxxxxx" failed (2: No such file or directory), client: xx.xx.xx.xx, server: xxxxx.xx, request: "xxxxxxx HTTP/1.1", host: "xxxxx.xx"

I am extracting fields with the following logstash filters:

grok {
match => {
    "message" => [
            "(?<logstamp>\h{3} \d{2} \d{2}:\d{2}:\d{2}) %{WORD:hostname} (?<source>[^\s]+) (?<timestamp>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) %{GREEDYDATA:error_message}" 
 ]
 }

date {
locale => "en"
match => [ "timestamp", "yyyy/MM/dd HH:mm:ss" ]
    }
}

mutate {
remove_field => [ "@version", "_score", "message", "host", "_type", "logstamp" ]
}

Based on http://grokdebug.herokuapp.com/, my syntax is sane.
I have two dates in the log line because the first one is when rsyslog received the line, and the second one is from nginx. What I want is to pass the second one to "timestamp".

The error I get in logstash is:

@metadata_accessors=#<LogStash::Util::Accessors:0x1d630482 @store={"path"=>"..."}, @lut={"[path]"=>[{"path"=>"..."}, 
"path"]}>, @cancelled=false>], :response=>{"create"=>{"_index"=>"...", "_type"=>"...", "_id"=>"...", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", 
"reason"=>"failed to parse [timestamp]", "caused_by"=>{"type"=>"illegal_argument_exception",
"reason"=>"Invalid format: \"2016/02/16 12:25:16\" is malformed at \"/02/16 12:25:16\""}}}}, :level=>:warn}

(I clipped the output to make it shorter)

EDIT: WORKING CONFIG
I ended up converting the timestamp from Nginx log to a more standard one (as seen in the ruby part), and using that one in the date match as @timestamp.

  grok {
match => {
    "message" => [
            "(?<logstamp>\h{3} \d{2} \d{2}:\d{2}:\d{2}) %{WORD:hostname} (?<source>[^\s]+) (?<ngxstamp>[^\s]+ [^\s]+) %{GREEDYDATA:error_message}"
     ]
    }
}

ruby {
code => "event['ngxstamp'] = event.timestamp.time.localtime.strftime('%Y-%m-%d %H:%M:%S')"
}

date {
match => [ "ngxstamp", "yyyy-MM-dd HH:mm:ss" ]
locale => "en"
}

mutate {
remove_field => [ "@version", "_score", "message", "host", "_type", "logstamp" ]
}

回答1:


Since the type of your timestamp field is strict_date_optional_time, the date pattern you should be using in your date filter should be

yyyy-MM-dd HH:mm:ss

instead of

yyyy/mm/dd HH:mm:ss

So:

  • Use dashes instead of slashes in the date part
  • use MM instead of mm for the months

There might still be an issue with the missing T between the date and time parts, since strict_date_optional_time mandates it, though.



来源:https://stackoverflow.com/questions/35433291/logstash-date-invalid-format

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!