Recently, I upgraded an internal ELK stack that was running version 7.x components to 8.x. This post outlines the process that was followed (steps might be different for other versions). Also, this post does not deep-dive into the process since the excellent Elastic docs are available, and they are regularly updated. Please consider this a process outline, rather than a guide.
This specific stack has been set up for enabling observability into application logs, and multiple other related logs, of which the average daily index size is at the north of 10 G owing to business requirements. The stack had been configured with a set of rollup jobs, transform jobs, and dashboards.
Sample ELK Stack Setup
Before the upgrade
When it comes to getting to know the deprecations etc, Elastic documentation regarding upgrades1 is the resource we should be looking into. While I too frequented the docs, I also happened to have the support of a great teammate2 who is well-versed in Elastic-fu! The docs are comprehensive, and thanks to the awesome community around Elastic products, you’ll most probably find more than one person that took the path you are trying to take, unless of course you’ve a Frankstein of a stack in your hands of course!
Also, refer the Elastic support matrix3 which include version compatibility information!
In my case, the docs recommended upgrading to the latest available version of ELK 7.x, which is 7.17.25 at the time of writing. The order of the component-wise upgrade is recommended as below (see Elastic docs for detiled info).
- Elasticsearch
- Kibana
- Logstash
- Filebeat
The upgrade process
Once I was on the latest 7.x version, it was about using the “Upgrade Assistant” wizard on Kibana for identifying known issues/incompatibilities against the expected version of 8.x. The wizard was really handy, and resolved most of the incompatibilities “automagically”, while I had to remove/reconfigure a couple indices/policies.
Following configuration updates in elasticsearch.yml and jvm.options(use jvm.options.d directory!) to be 8.x-ready and once “Upgrade Assistant” indicated that our stack is ready for the expected version, I first upgraded each of our Elasticsearch instances running on idividual VMs on RHEL with “rpm -Uvh elasticsearch-8.x.rpm”. After upgrading all instances in the Elasticsearch cluster without issues, I proceeded with the Kibana and Logstash instances in a similar manner. Finally Filebeat was upgraded, but reverted to 7.17.25 since a difference in the Filebeat-appended metadata, which should have been identified earlier. However, since the version configurations are compatibile, Filebeat was kept at that version.
Following the upgrade
Since rollup jobs are deprecated in 8.x, transform jobs were used in their place. Downsampling, which replaces rollup jobs, does not cover all functionalities offered by the preceding feature.
As with any major version upgrade, while we have cluster monitoring/alerting set up, stats and health was closely monitored for a period of time after the task was completed. And the relevant information documented for internal DevOps members.
As already noted, please refer the frequently updated Elastic docs for detailed info on the Elastic stack upgrade process.