How to stop logstash from creating a default mapping in ElasticSearch

Question

I am using logstash to feed logs into ElasticSearch. I am configuring logstash output as:

input {
file {
            path => \"/tmp/foo.log\"
            code         


        

Answer 1

You will need a mapping to store data in Elasticsearch and to search on it - that's how ES knows how to index and search those content types. You can either let logstash create it dynamically or you can prevent it from doing so and instead create it manually.

Keep in mind you cannot change existing mappings (although you can add to them). So first off you will need to delete the existing index. You would then modify your settings to prevent dynamic mapping creation. At the same time you will want to create your own mapping.

For example, this will create the mappings for the logstash data but also restrict any dynamic mapping creation via "strict":

$ curl -XPUT 'http://localhost:9200/4glogs/logs/_mapping' -d '
{
    "logs" : {
        "dynamic": "strict",
        "properties" : {
            "@timestamp": {
                "type": "date",
                "format": "dateOptionalTime"
                    },
            "@version": {
                "type": "string"
                    },
             "message": {
                "type": "string"
                    }
        }
    }
}
'

Keep in mind that the index name "4glogs" and the type "logs" need to match what is coming from logstash.

For my production systems I generally prefer to turn off dynamic mapping as it avoids accidental mapping creation.

The following links should be useful if you want to make adjustments to your dynamic mappings:

https://www.elastic.co/guide/en/elasticsearch/guide/current/dynamic-mapping.html

http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/custom-dynamic-mapping.html

http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/dynamic-mapping.html

Answer 2

logs in this case is the index_type. If you don't want to create it as logs, specify some other index_type on your elasticsearch element. Every record in elasticsearch is required to have an index and a type. Logstash defaults to logs if you haven't specified it.

There's always an implicit mapping created when you insert records into Elasticsearch, so you can't prevent it from being created. You can create the mapping yourself before you insert anything (via say a template mapping).

The setting manage_template of false just prevents it from creating the template mapping for the index you've specified. You can delete the existing template if it's already been created by using something like curl -XDELETE http://localhost:9200/_template/logstash?pretty

Answer 3

Index templates can help you. Please see this jira for more details. You can create index templates with wildcard support to match an index name and put your default mappings.