How to filter input data of logstash based on date filed?

我是研究僧i 提交于 2019-12-13 13:34:54

问题


here is my twitter input tweets

"_source": {
"created_at": "Wed Aug 10 06:42:48 +0000 2016",
"id": 763264318242783200,
"timestamp_ms": "1470811368891",
"@version": "1",
"@timestamp": "2016-08-10T06:42:48.000Z"
}

and my logstash config file which include twitter input plugin filter and output

input {
twitter {
consumer_key => "lvvoeonCRBOHsLAoTPbion9sK"
consumer_secret => "GNHOFzErJhuo0bNq38JUs7xea2BOktMiLa7tunoGwP0oFKCHrY"
oauth_token => "704578110616936448-gfeSklNrITu7fHIZgjw3nwoZ1S0l0Jl"
oauth_token_secret => "IHiyRJRN09jjdUTGrnesALw4DRle35WyX7pdnI3CtEnJ5"
keywords => [ "afghanistan", "TOLOnews", "kabul", "police"]
full_tweet => true
}
}
filter {
    date {
      match => ["timestamp" , "MMM d YYY HH:mm:ss", "ISO8601"]
  }
 }
output {
   stdout { codec => dots }
    elasticsearch {
        hosts => "10.20.1.123"
        index => "twitter_news"
        document_type => "tweets"
    }
}

I want to just get new tweets for example today date is 2016-11-16, then I just want to get tweets that have @timestamp= 2016-11-16 not @timestamp= 2016-11-15 or past days tweets, but with this configuration i get past tweets as well, any one help me to how do this ?


回答1:


the idea here is to use ruby code in logstash config. I propose to use timestamp_ms for comparing date.

  1. First need to convert timestamp_ms to integer
  2. Add today timestamp in ms with ruby
  3. Compare timestamps Here is an example:

    mutate {
        convert => {
            "timestamp_ms" => "integer"
        }
    }
    ruby {
        code => "
            t = Time.now
            today_ymd = t.strftime('%Y%m%d')
            today_timestamp_ms = DateTime.parse(today_ymd).to_time.to_i*1000
            event['@metadata']['today_timestamp_ms'] = today_timestamp_ms
        "
    }
    
    
    if [timestamp_ms] < [@metadata][today_timestamp_ms] {
    ## past days events
        mutate {
            add_field => { "test" => "past days events"  }
        }
    } else {
    # today events
        mutate {
            add_field => { "test" => "today events"  }
        }
    
    }
    


来源:https://stackoverflow.com/questions/40624141/how-to-filter-input-data-of-logstash-based-on-date-filed

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!