Drop log messages containing a specific string

Question

So I have log messages of the format :

[INFO]  <blah.blah>       2016-06-27 21:41:38,263 some text
[INFO]  <blah.blah>       2016-06-28 18:41:38,262 some other text

Now I want to drop all logs that does not contain a specific string "xyz" and keep all the rest. I also want to index timestamp.

grokdebug is not helping much.

This is my attempt :

input {
    file {
         path => "/Users/username/Desktop/validateLogconf/logs/*"
      start_position => "beginning"

   }
}

filter {

  grok {
      match => {
      "message" => '%{SYSLOG5424SD:loglevel}  <%{JAVACLASS:job}>       %{GREEDYDATA:content}'
      }
  }

  date {
    match => [ "Date", "YYYY-mm-dd HH:mm:ss" ]
    locale => en
  }

}

output {
  stdout {
codec => plain {
                        charset => "ISO-8859-1"
                }

}
    elasticsearch {
        hosts => "http://localhost:9201"
        index => "hello"

  }
}

I am new to grok so patterns above might not be making sense. Please help.

Answer 1

To drop the message that does not contain the string xyz:

if ([message] !~ "xyz") {
    drop { }
}

Your grok pattern is not grabbing the date part of your logs.
Once you have a field from your grok pattern containing the date, you can invoque the date filter on this field.
So your grok filter should look like this:

grok {
    match => {
        "message" => '%{SYSLOG5424SD:loglevel}  <%{JAVACLASS:job}>       %{TIMESTAMP_ISO8601:Date} %{GREEDYDATA:content}'
    }
}

I added a part to grab the date, which will be in the field Date. Then you can use the date filter:

date {
    match => [ "Date", "YYYY-mm-dd HH:mm:ss,SSS" ]
    locale => en
}

I added the ,SSS so that the format match the one from the Date field. The parsed date will be stored in the @timestamp field, unless specified differently with the target parameter.

Answer 2

to check if your message contains a substring, you can do:

if [message] =~ "a" {
   mutate {
      add_field => { "hello" => "world" }
   }
}

So in your case you can use the if to invoke the drop{} filter, or you can wrap your output plugin in it.

To parse a date and write it back to your timestamp field, you can use something like this:

date {
    locale => "en"
    match => ["timestamp", "ISO8601"]
    timezone => "UTC"
    target => "@timestamp"
    add_field => { "debug" => "timestampMatched"}
}

This matches my timestamp in:

• Source field: "timestamp" (see match)
• Format is "ISO...", you can use a custom format that matches your timestamp
• timezone - self explanatory
• target - write it back into the event's "@timestamp" field
• Add a debug field to check that it has been matched correctly

Hope that helps,

Artur