Removing old indices in elasticsearch

Question

I have the many of my logs indexed in logstash-Year-Week format. That is if i want to delete indices older than a few weeks, how can I achieve that in elasticsearch. Is ther

Answer 1

If you are using elasticsearch version 5.x then you need to install the curator version 4.x. You can see the version compatibility and installation steps from the documentation

Once installed. Then just run the command

curator --config path/config_file.yml [--dry-run] path/action_file.yml

Curator provides a dry-run flag to just output what Curator would have executed. Output will be in your log file which you have defined in config.yml file. If not logging key defined in config_file.yml then currator will output to console. To delete the indices run the above command without --dry-run flag

The configuration file config_file.yml is

---
client:
  hosts:
   - 127.0.0.1
  port: 9200
logging:
  loglevel: INFO
  logfile: "/root/curator/logs/actions.log"
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

The action file action_file.yml is

---
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 7 days (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 7
      exclude:

If you want to delete the indices weekly, monthly, etc automatically. Then just write the bash script like

#!/bin/bash
# Script to delete the log event indices of the elasticsearch weekly

#This will delete the indices of the last 7 days
curator --config /path/config_file.yml /path/action_file.yml

Put a shell script in one of these folders: /etc/cron.daily, /etc/cron.hourly, /etc/cron.monthly or /etc/cron.weekly and your job is done.

NOTE: Make sure to use the correct indentation in your configuration and action files. Otherwise it will not work.

Answer 2

In my case removing old indices was mandatory since I had Upgraded to 7.5 version from 5.X,

So I followed simple step to clear the indices.

rm -rf /var/lib/elasticsearch/nodes/0/indices/*

Answer 3

you can use curl

 curl -X DELETE http://localhost:9200/filebeat-$(date +"%Y.%m.%d" -d "last Month")

this must to add this command to xxx.sh, and you can create crontab. crontab -e

00 00 * * * /etc/elasticsearch/xxx.sh

this cron will running everyday at 12pm and it will remove old log.

Answer 4

Take a look at Curator, a tool developed specially for this kind of use case.

A sample command, for the documentation:

curator --host 10.0.0.2 delete indices --older-than 30 --time-unit days \
   --timestring '%Y.%m.%d'

Answer 5

As of elasticsearch 6.6, Index Lifecycle Management comes included with basic (free) versions elasticsearch, and accomplishes what Curator used to, but in a more graceful way.

The steps below are reproduced without permission from Martin Ehrnhöfer's excellent and concise blog post.

Assumptions (heads up to the copy-pasters):

• Your elasticsearch server is accessible at http://elasticsearch:9200
• You want your indices to be purged after thirty days (30d)
• Your policy name will be created as cleanup_policy
• Your filebeat index names begin with filebeat-
• Your logstash index names begin with logstash-

1. Create a policy that deletes indices after one month

curl -X PUT "http://elasticsearch:9200/_ilm/policy/cleanup_policy?pretty" \
     -H 'Content-Type: application/json' \
     -d '{
      "policy": {                       
        "phases": {
          "hot": {                      
            "actions": {}
          },
          "delete": {
            "min_age": "30d",           
            "actions": { "delete": {} }
          }
        }
      }
    }'

2. Apply this policy to all existing filebeat and logstash indices

curl -X PUT "http://elasticsearch:9200/logstash-*/_settings?pretty" \
     -H 'Content-Type: application/json' \
     -d '{ "lifecycle.name": "cleanup_policy" }'
curl -X PUT "http://elasticsearch:9200/filebeat-*/_settings?pretty" \
     -H 'Content-Type: application/json' \
     -d '{ "lifecycle.name": "cleanup_policy" }'

3. Create a template to apply this policy to new filebeat and logstash indices

curl -X PUT "http://elasticsearch:9200/_template/logging_policy_template?pretty" \
     -H 'Content-Type: application/json' \
     -d '{
      "index_patterns": ["filebeat-*", "logstash-*"],                 
      "settings": { "index.lifecycle.name": "cleanup_policy" }
    }'

Answer 6

I use a bash script, just change the 30 with the # of days you want to keep

#!/bin/bash

# Zero padded days using %d instead of %e
DAYSAGO=`date --date="30 days ago" +%Y%m%d`
ALLLINES=`/usr/bin/curl -s -XGET http://127.0.0.1:9200/_cat/indices?v | egrep logstash`

echo
echo "THIS IS WHAT SHOULD BE DELETED FOR ELK:"
echo

echo "$ALLLINES" | while read LINE
do
  FORMATEDLINE=`echo $LINE | awk '{ print $3 }' | awk -F'-' '{ print $2 }' | sed 's/\.//g' ` 
  if [ "$FORMATEDLINE" -lt "$DAYSAGO" ]
  then
    TODELETE=`echo $LINE | awk '{ print $3 }'`
    echo "http://127.0.0.1:9200/$TODELETE"
  fi
done

echo
echo -n "if this make sence, Y to continue N to exit [Y/N]:"
read INPUT
if [ "$INPUT" == "Y" ] || [ "$INPUT" == "y" ] || [ "$INPUT" == "yes" ] || [ "$INPUT" == "YES" ]
then
  echo "$ALLLINES" | while read LINE
  do
    FORMATEDLINE=`echo $LINE | awk '{ print $3 }' | awk -F'-' '{ print $2 }' | sed 's/\.//g' `
    if [ "$FORMATEDLINE" -lt "$DAYSAGO" ]
    then
      TODELETE=`echo $LINE | awk '{ print $3 }'`
      /usr/bin/curl -XDELETE http://127.0.0.1:9200/$TODELETE
      sleep 1
      fi
  done
else 
  echo SCRIPT CLOSED BY USER, BYE ...
  echo
  exit
fi