Parse a log using Losgtash

问题

I am using Logstash to parse a log file. A sample log line is shown below.

2011/08/10 09:51:34.450457,1.048908,tcp,213.200.244.217,47908, ->,147.32.84.59,6881,S_RA,0,0,4,244,124,flow=Background-Established-cmpgw-CVUT

I am using following filter in my confguration file.

 grok {
       match => ["message","%{DATESTAMP:timestamp},%{BASE16FLOAT:value},%{WORD:protocol},%{IP:ip},%{NUMBER:port},%{GREEDYDATA:direction},%{IP:ip2},%{NUMBER:port2},%{WORD:status},%{NUMBER:port3},%{NUMBER:port4},%{NUMBER:port5},%{NUMBER:port6},%{NUMBER:port7},%{WORD:flow}" ]
    }

It works well for error-free log lines. But when I have a line like below, it fails. Note that the second field is missing.

2011/08/10 09:51:34.450457,,tcp,213.200.244.217,47908, ->,147.32.84.59,6881,S_RA,0,0,4,244,124,flow=Background-Established-cmpgw-CVUT

I want to put a default value in there in my output Json object, if a value is missing. how can I do that?

回答1:

Use (%{BASE16FLOAT:value})? for second field to make it optional - ie. regex ()? .

Even if the second field is null the grok will work.

So entire grok look like this:

%{DATESTAMP:timestamp},(%{BASE16FLOAT:value})?,%{WORD:protocol},%{IP:ip},%{NUMBER:port},%{GREEDYDATA:direction},%{IP:ip2},%{NUMBER:port2},%{WORD:status},%{NUMBER:port3},%{NUMBER:port4},%{NUMBER:port5},%{NUMBER:port6},%{NUMBER:port7},%{WORD:flow}

回答2:

Use it in your conf file. Now, if value field is empty it will omit it in response.

input {
   stdin{
   }
}
filter {

grok {
       match => ["message","%{DATESTAMP:timestamp},%{DATA:value},%{WORD:protocol},%{IP:ip},%{NUMBER:port},%{GREEDYDATA:direction},%{IP:ip2},%{NUMBER:port2},%{WORD:status},%{NUMBER:port3},%{NUMBER:port4},%{NUMBER:port5},%{NUMBER:port6},%{NUMBER:port7},%{WORD:flow}" ]
    }

}
output {
  stdout {
        codec => rubydebug
  }
}

来源：https://stackoverflow.com/questions/35429265/parse-a-log-using-losgtash