Query fields in Kibana with RegEx

问题

I need to search in Kibana Logs for fields with a specific content. The field is "message", that looks like this:

11.111.72.58 - - [26/Nov/2020:08:44:23 +0000] "GET /images/image.jpg HTTP/1.1" 200 123456 "https://website.com/questionnaire/uuid/result" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.14 (KHTML, like Gecko) Version/14.0.1 Safari/605.1.14" "5.158.163.231"

This field contains URIs, for example here "https://website.com/questionnaire/uuid/result". How can I search for specific URIs in that field? I need to get all Logs, where the field "message" contains "https://website.com/questionnaire/someUUID*/result" or where the URI is exactly "https://website.com/"

I've tried with Lucene:

message:/https://.+/result/

nothing found

message:https.*\result

URIs with "https: at the beginning found, but also returns URIs without "result" at the end

message : "https://website.com/questionnaire" AND message : "result"

This works, but this would also work, if "result" would not be related to the URI, but would just stay alone at the end of the "message"-field. And I need something, that would really query those URIs between " ". I need to visualise the amount of requests for each URI with Kibana later. So I think I need to use Lucene or Query DSL. Any ideas?

回答1:

This is a good use case for the new wildcard field type (introduced in 7.9), which allows you to better search within potentially long strings.

If you declare your message field as wildcard like this:

PUT test 
{
  "mappings": {
    "properties": {
      "message": {
        "type": "wildcard"
      }
    }
  }
}

And then index your documents

PUT test/_doc/1
{
  "message": """11.111.72.58 - - [26/Nov/2020:08:44:23 +0000] "GET /images/image.jpg HTTP/1.1" 200 123456 "https://website.com/questionnaire/uuid/result" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.14 (KHTML, like Gecko) Version/14.0.1 Safari/605.1.14" "5.158.163.231"
  """
}

You can then run wildcard searches (even with leading wildcards which are discouraged to run on normal keyword fields) and find your document easily.

GET test/_search
{
  "query": {
    "wildcard": {
      "message": {
        "value": "*https*uuid*"
      }
    }
  }
}

来源：https://stackoverflow.com/questions/65021291/query-fields-in-kibana-with-regex

标签

regex

ElasticSearch

lucene

kibana