Retrieving RESTful GET parameters in logstash

Question

I am trying to get logstash to parse key-value pairs in an HTTP get request from my ELB log files.

the request field looks like http://aaa.bbb/get?a=1&b=2

I'd like there to be a field for a and b in the log line above, and I am having trouble figuring it out.

My logstash conf (formatted for clarity) is below which does not load any additional key fields. I assume that I need to split off the address portion of the URI, but have not figured that out.

input {
    file {
        path => "/home/ubuntu/logs/**/*.log"
        type => "elb"
        start_position => "beginning"
        sincedb_path => "log_sincedb"
    }
}
filter {
    if [type] == "elb" {
        grok {
            match => [ "message", "%{TIMESTAMP_ISO8601:timestamp} 
%{NOTSPACE:loadbalancer} %{IP:client_ip}:%{NUMBER:client_port:int}
%{IP:backend_ip}:%{NUMBER:backend_port:int} 
%{NUMBER:request_processing_time:float}
%{NUMBER:backend_processing_time:float} 
%{NUMBER:response_processing_time:float} 
%{NUMBER:elb_status_code:int}
%{NUMBER:backend_status_code:int} 
%{NUMBER:received_bytes:int} %{NUMBER:sent_bytes:int} 
%{QS:request}" ]
        }
        date {
            match => [ "timestamp", "ISO8601" ]
        }
        kv {
            field_split => "&?"
            source => "request"
            exclude_keys => ["callback"]
        }
    }
}


output {
    elasticsearch { host => localhost }
}

Answer 1

kv will take a URL and split out the params. This config works:

input {
    stdin { }
}

filter {
    mutate {
            add_field => { "request" => "http://aaa.bbb/get?a=1&b=2" }
    }

    kv {
            field_split => "&?"
            source => "request"
    }
}

output {
    stdout {
            codec => rubydebug
    }
}

stdout shows:

{
   "request" => "http://aaa.bbb/get?a=1&b=2",
         "a" => "1",
         "b" => "2"
}

That said, I would encourage you to create your own versions of the default URI patterns so that they set fields. You can then pass the querystring field off to kv. It's cleaner that way.

UPDATE:

For "make your own patterns", I meant to take the existing ones and modify them as needed. In logstash 1.4, installing them was as easy as putting them in a new file the 'patterns' directory; I don't know about patterns for >1.4 yet.

MY_URIPATHPARAM %{URIPATH}(?:%{URIPARAM:myuriparams})?
MY_URI %{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{MY_URIPATHPARAM})?

Then you could use MY_URI in your grok{} pattern and it would create a field called myuriparams that you could feed to kv{}.