如何将正在运行的 Elasticsearch 旧实例升级到新版本？

Question

Essentially I cannot find documents or resources that explains the procedure of upgrading a running Elasticsearch instance into the current version.

Please help me out in a few scenarios:

1. If I am running an Elasticsearch instance in a single server, how do I upgrade the instance and not lose data?

2. If I am running multiple Elasticsearch instances in a number of servers, how do I keep my operations running, while I upgrade my Elasticsearch instances without losing data?

If there are proper procedures or explanations on this it will greatly help my understanding and work. Thanks!

Answer 1

1. All node data is stored in elasticsearch data directory. It's data/cluster_name/nodes by default in elasticsearch home. So, in general, as long as data directory is preserved and config files in the new version are compatible with the old version, the new instance should have the same data as the old one. Please note that some releases have special additional requirements outlined in release notes . For example, upgrade to 0.19 from 0.18 requires issuing a full flush of all the indices in the cluster.

2. There is really no good way to accomplish this. Nodes are communicating using binary protocol that is not backward compatible. So, if protocol in the new version changes, old nodes and new nodes are unable to understand each other. Sometimes it's possible to mix nodes with different minor versions within the same cluster and do rolling upgrade. However, as far as I understand, there is no explicit guarantee on compatibility between nodes even within minor releases and major releases always require require full cluster restart. If downtime during full cluster restart is not an option, a nice technique by DrTech might be a solution.

Answer 2

There is a lot more information about ElasticSearch upgrade these days than it used to be.

Here are my usual steps when upgrading ElasticSearch:

1. Backup the data : Snapshot and Restore

2. Upgrade Guide : Upgrading ElasticSearch

The main idea is that you shut down one instance of the ES cluster at a time, upgrade the ES version on that instance node, and bring it up again so it can join back the cluster.

In brief, here are the important steps:

1. Disable Shard reallocation

   curl -XPUT localhost:9200/_cluster/settings -d '{ "transient" : { "cluster.routing.allocation.enable" : "none" } }'

2. Shutdown the instance:

   curl -XPOST 'http://localhost:9200/_cluster/nodes/_local/_shutdown'

3. Install the new ElasticSearch version on the host and Start it.

4. Enable shard re-allocation:

   curl -XPUT localhost:9200/_cluster/settings -d '{ "transient" : { "cluster.routing.allocation.enable" : "all" } }'

5. Watch cluster go from yellow state to green with:

curl -X GET http://localhost:9200/_cat/health?v // monitors the overal cluster state

curl -X GET http://localhost:9200/_cat/nodes?v // verify that the new node joined the cluster

curl -X GET http://localhost:9200/_cat/shards?v // see shards being started, initialized and relocated

6. Repeat for the next node.

In the terms of ordering, update first the master nodes , then data nodes, then load-balancing/client nodes.

Answer 3

It should be worth mentioning that there is now documentation on doing this upgrade but it doesn't rank very highly on search results:

Upgrade Elasticsearch

as well as a breaking changes document:

Breaking Changes

Answer 4

The other answers are a little dated based off what exists now, so this new information will supplement upgrading newer versions of Elasticsearch, like upgrading 6.x to 7.x, or 5.x to 6.x. There are two main options for an Elasticsearch upgrade: rolling upgrade or full cluster restart .

The rolling upgrade allows nodes to be upgraded one at a time so service is not interrupted. On the other hand, the full cluster restart requires every node to be shut down, upgraded, and then brought back up. This means there will be downtime between upgrades that must be accounted for.

This is much easier to do compared to a few years ago when the only viable option was a snapshot & restore.

Answer 5

Here are two articles about this question that I really liked!

• https://www.freecodecamp.org/news/how-to-migrate-from-elasticsearch-1-7-to-6-8-with-zero-downtime/
• https://blog.devartis.com/how-we-upgraded-a-10-tb-elasticsearch-cluster-from-1-7-to-6-7-466f6a33a6ca

Basically the approach is to start a new cluster (with the latest ElasticSearch version you want), keep your data synced by changing your code to make it index a new / updated entity in both (old and new) clusters and start a full reindex on to your new cluster.

At this time you should have two perfectly synced clusters regarding your data. That allows you to start plugging your read features to the new cluster, see how it goes, probably fix the code making the ES requests regarding the breaking changes (between your current ES version and the new one you want to use), easily rollback to the old cluster if needed.

Once you are all good you can start plugging your write / updates features... and eventually after a few weeks or months turn off the old cluster.

It's really flexible but all depends of the usecase / timeline your are facing. The main possible issue is : does the mapping of the old cluster will be fully compatible with the new one ?

Hope it helps! I wish you a great day :)

Answer 6

If you are running Ubuntu or Debian based Linux, here is an Ansible script to do a rolling upgrade, as long as you are not upgrading between major versions.

Minor versions such as 1.3 -> 1.4.3 are ok

Major versions such as 0.8 -> 1.4.3 will not work.

https://github.com/ekhoinc/ansible-examples/blob/master/elasticsearch-rolling-upgrade.yml

It could easily be modified to work with RHEL based linux (just 2 lines to change)