Do you have elasticsearch somewhere in your infra? I do and that is to say extensively but this post is not about how gorgeous it is to use but rather how ugly it can be after the first six months of use or the year,we use the tool in both graylog and elk-stack. However if ever you used it, you know that it is quite a bit of headache to manage and to monitor it i.e the administration of the size on disk,various indices administration and snapshotting.
Lets be honest, the graylog interface has got it right on retention/rotation on the system/indicdes menu;a thumps up for graylog guys. This is where elk-stack fails poorly, particularly with the newest and brightest of the all: 5.6 version.
I admit to spend many days and weeks in search of a better tool than simply going to “curl -XDELETE http://localhost:9200/the_target_indices -u “username:passwprd” “.
There is a tool called the elasticsearch-HQ where by a tweak on the elasticsearch.yml file and adding the below entries:
you may connect to it in a browser and encounter massive information about any part of your elasticsearch. However it does not allow you any administration…it is rather a pretty GUI for the things under the hood.
This is where we need to turn to a python tool called elasticsearch-curator that does the job. It is acquired by the elastic team lately and maintained by them which shows the importance of such tool.you can see the full documentation in here.
Here is the impatient guide in installing and running it on the elasticsearch node.
a. Install pip
yum isntall python-pip python-virtualenv
we need python virtual env to keep the host env seperate from where we run curator as versioning of heck of a lot of python tools do not get into your hair.
b. create a dir
c. create the environment
d. get to the environment
e.Install elasticsearch curator
pip install elasticsearch-curator
f. check the version and note that the compatibility of the version of curator with your elasticsearch in here
Now rest of this article assumes that you successfully got to the stage where you have the latest curator i.e 5.3 and the latest elasticsearch i.e 5.6 which happens to be secured by the x-pack. Basically you need two files preferably in the same dir as yor working env for the ease of use and reference, which are config.yml and action_delete.yml
- 127.0.0.1 #this is the elasticsearch listening port interface
-action_delete.yml example(Caution:please do read the descriptions and change them accordingly):
# want to use this action as a template, be sure to set this to False after
# copying it.
Delete indices older than 30 days (based on index name), for logstash-
prefixed indices. Ignore the error if the filter does not result in an
actionable list of indices (ignore_empty_list) and exit cleanly.
- filtertype: pattern
value: metricbeat- #trying to filter only on the *metricbeat* indices
- filtertype: age
Now lets get dirty: first things first try to run the code to get indices and make sure all is good:
curator_cli --config config.yml show_indices
It should return you all the indices…and now with the action:
curator --config config.yml action_delete_metricbeat.yml
which will delete indices older than 30 days.