Indices Red/Dangling when upgrading Elasticsearch cluster from 2.x to 5.x in docker

Question

I manage a small ELK stack which resides on one docker host with a 3 node elasticsearch cluster, a master, a client node, and a data node each running elasticsearch 2.4.x. Each of these nodes had the same host directory bind mounted as the elasticsearch data directory even though only the data node needed it.

While testing the upgrade path to 5.x I was running into a very strange issue. The cluster would come back up, but would not initialize any of the 2.x created indices, throwing errors:

[oegDanglingIndicesState] [elastic-data] [[logstash-2017.02.01/pBco8d7dQAqmZoI37vUIOQ]] dangling index exists on local file system, but not in cluster metadata, auto import to cluster state

The indices would never initialize and remain red. The stack would create new indices fine and if I deleted these indices the system would work perfectly fine, but that data loss is definitely sub-optimal if I was going to do this on a production system.

Answer 1

The fact the the master and client nodes had been mounting the data directory turned out to be the cause of this issue. Elasticsearch 5.x enforces a default limit of 1 node to a data directory and while the master and client 2.x nodes had not been actively data managing, they had affected the folder structuring. I was able to get a clean upgrade with all green indices by first removing the bind mounts from master and client on the 2.x cluster and letting that sort itself out, then upgrading to 5.x. Hope this helps anybody else who runs into this issue.