Logstash mutate add all fields from json

问题

I am using a Logstash plugin (logstash-input-rethinkdb). This plugin grabs all the edits in the database and outputs an json object containing the following structure:

{
      "db":"itjobs",
      "table":"countries", 
      "old_val":null,
      "new_val":{
           "code":"USA3",
           "country":"USA3",
           "id":"7c8c9e4e-aa37-48f1-82a5d624cde4a3a0"
      },
      "@version":"1",
      "@timestamp":"2016-12-19T19:54:08.263Z"
 }

I insert this doc in elasticsearch.
But the problem is that in elastic i get the same structure with -> new_val:{code:''}

I need to do a filter or something that will extract everything from new_val and add it to the root

I tried the filter json but this get a string and the plugin already outputs json

After edit i need it to look like:

{
      "db":"itjobs",
      "table":"countries", 
      "code":"USA3",
      "country":"USA3",
      "id":"7c8c9e4e-aa37-48f1-82a5d624cde4a3a0"
      "@version":"1",
      "@timestamp":"2016-12-19T19:54:08.263Z"
 }

回答1:

You can use the ruby filter:

filter {
    ruby {
        code => "
            event['new_val'].each {|k, v|
                event[k] = v
            }
            event.remove('new_val')
        "
    }
}

Tested with Logstash 2.2, it should be different for version 5+.

回答2:

This works after they changed class Logstash::Event to API: https://www.elastic.co/guide/en/logstash/current/event-api.html

filter {
  ruby {
    code => "
      event.get('new_val').each {|k, v|
        event.set(k,v)
      }
      event.remove('new_val')
    "
  }
}

来源：https://stackoverflow.com/questions/41230766/logstash-mutate-add-all-fields-from-json

标签

ElasticSearch

logstash

rethinkdb