How to delete elastic search indices periodically?

Question

I have created indices on daily basis to store the search history and i am using those indices for the suggestions in my applciation, which helps me to suggest based on history as well.

now i have to maintain only last 10 days of history. So is there any feature in Elastic search that allows me to create and delete indices periodically?

Answer 1

The only thing I can think of is using data math: https://www.elastic.co/guide/en/elasticsearch/reference/current/date-math-index-names.html

In sense you can do this:

DELETE <logs-{now%2Fd-10d}>

This does not work nice in curl though due to url encoding. You can do something like this in curl:

curl -XDELETE 'localhost:9200/<logs-%7Bnow%2Fd-10d%7D>'

Both examples remove the index that is 10 days old. It does not help you in deleting indices older than 10 days, don't think that is possible. And their is no trigger or something in elasticsearch.

So I would stick to a cron job in combination with curator, but you do have this option to go with as well.

Answer 2

I don't know if elasticsearch has a built-in feature like that but you can achieve what you want with curator and a cron job.

An example curator command is:

Curator 3.x syntax [deprecated]:

curator --host <IP> delete indices --older-than 10 --prefix "index-prefix-" --time-unit days  --timestring '%Y-%m-%d'

Curator 5.1.1 syntax:

curator_cli --host <IP> --port <PORT> delete_indices --filter_list '[{"filtertype":"age","source":"creation_date","direction":"older","unit":"days","unit_count":10},{"filtertype":"pattern","kind":"prefix","value":"index-prefix-"}]'

Run this command daily with a cron job to delete indices older than 10 days whose names start with index-prefix- and that live on the Elasticsearch instance at <IP>:<PORT>.

For more curator command-line options, see: https://www.elastic.co/guide/en/elasticsearch/client/curator/current/singleton-cli.html

For more on cron usage, see: http://kvz.io/blog/2007/07/29/schedule-tasks-on-linux-using-crontab/