Logstash: Keeping a value across events

Question

I have a date that is only present once in every log file and I am trying to add this date to all following events after it has been matched once, making it act like a globa

Answer 1

Maybe you should use the official aggregate filter for this, since memorize is not official and will not work with Logstash >2.0.

It would go like this:

# same as what you have now
grok {
    patterns_dir => "[...]"
    match => [ "message", "%{DATELINE}" ]
    tag_on_failure => [ "not_date_line" ]

}
# add a fictional taskId field to correlate all lines
mutate {
   add_field => { "taskId" => "all" }
}

# if we're processing the first line, remember the date
if "not_date_line" not in [tags] {
    aggregate {
        task_id => "%{taskId}"
        code => "map['mydate'] = event['mydate']"
    }
} 
# if we're processing the next lines, add the date
else {
    aggregate {
        task_id => "%{taskId}"
        code => "event['mydate'] = map['mydate']"
        map_action => "update"
        timeout => 0
    }
}

All your events will then have a mydate field with the date that was on the first log line.