Logstash Dynamically assign template

问题

I have read that it is possible to assign dynamic names to the indexes like this:

elasticsearch {
            cluster => "logstash"
            index => "logstash-%{clientid}-%{+YYYY.MM.dd}"
    }

What I am wondering is if it is possible to assign the template dynamically as well:

elasticsearch {
            cluster => "logstash"
            template => "/etc/logstash/conf.d/%{clientid}-template.json"
    }

Also where does the variable %{clientid} come from?

Thanks!

回答1:

Full disclosure: I am a Logstash developer at Elastic

You cannot dynamically assign a template because templates are uploaded only once, at Logstash initialization. Without the flow of traffic, deterministic variable completion does not happen. Since there is no traffic flow during initialization, there is nothing there which can "fill in the blank" for %{clientid}.

It is also important to remember that Elasticsearch index templates are only used when a new index is created, and so it is that templates are not uploaded every time a document reached the Elasticsearch output block in Logstash--can you imagine how much slower it would be if Logstash had to do that? If you intend to have multiple templates, they need to be uploaded to Elasticsearch before any data gets sent there. You can do this with a script of your own making using curl and Elasticsearch API calls. This also permits you to update templates without having to restart Logstash. You could run the script any time before index rollover, and when the new indices get created, they'll have the new template settings.

Logstash can send data to a dynamically configured index name, just as you have above. If there is no template present, Elasticsearch will create a best-guess mapping, rather than what you wanted. Templates can and ought to be completely independent of Logstash. This functionality was added for an improved out-of-the-box experience for brand new users. The default template is less than ideal for advanced use cases, and Logstash is not a good tool for template management if you have more than one index template.

回答2:

After some testing and feedback from other users, thanks Ben Lim, it seems this is not possible to do so far. The closest thing would be to do something like this:

    if [type] == "redis-input" {
            elasticsearch {
                    cluster => "logstash"
                    index => "%{type}-logstash-%{+YYYY.MM.dd}"
                    template => "/etc/logstash/conf.d/elasticsearch-template.json"
                    template_name => "redis"
            }
    } else if [type] == "syslog" {
            elasticsearch {
                    cluster => "logstash"
                    index => "%{type}-logstash-%{+YYYY.MM.dd}"
                    template => "/etc/logstash/conf.d/syslog-template.json"
                    template_name => "syslog"
            }
    }

来源：https://stackoverflow.com/questions/26724871/logstash-dynamically-assign-template

标签

logstash