Missing rollover_alias index setting in OpenSearch

Question

I am trying to setup index rollover in OpenSearch with simple min_doc_count condition, but I am getting "message": "Missing rollover_alias index setting [index=app_logs-000002]" error. I have a rollover alias called app_logs , and also have the following policy (for demo purpose it is dummy to rollover after 3 documents) attached to indexes:

PUT _plugins/_ism/policies/rollover_policy
{
    "policy": {
    "description": "Rollover policy",
    "default_state": "rollover",
    "states": [
        {
            "name": "rollover",
            "actions": [
                {
                    "rollover": {
                        "min_doc_count": 3
                    }
                }
            ],
            "transitions": []
        }
    ],
    "ism_template": [
        {
            "index_patterns": [
                "app_logs-*"
            ]
        }
    ]
    }
}

GET _cat/aliases :

app_logs                                   app_logs-000001                                        - - - false
app_logs                                   app_logs-000002                                        - - - true

GET _cat/indices :

yellow open app_logs-000002               V4j0gxaYTcqoQZvtd0u2zc 1 1    6 0  4.1kb  4.1kb
yellow open app_logs-000001               AnPjlOq6Q5We411z2q_YpQ 1 1    5 0 18.8kb 18.8kb
...

When doing GET _opendistro/_ism/explain/app_logs-000002?pretty I get:

{
  "app_logs-000002" : {
    "index.plugins.index_state_management.policy_id" : "rollover_policy",
    "index.opendistro.index_state_management.policy_id" : "rollover_policy",
    "index" : "app_logs-000002",
    "index_uuid" : "V4j0gxaYTcqoQZvtd0u2zc",
    "policy_id" : "rollover_policy",
    "policy_seq_no" : -2,
    "policy_primary_term" : 0,
    "rolled_over" : false,
    "index_creation_date" : 1659299029428,
    "state" : {
      "name" : "rollover",
      "start_time" : 1659299410303
    },
    "action" : {
      "name" : "rollover",
      "start_time" : 1659424192817,
      "index" : 0,
      "failed" : true,
      "consumed_retries" : 3,
      "last_retry_time" : 1659424804833
    },
    "step" : {
      "name" : "attempt_rollover",
      "start_time" : 1659424192817,
      "step_status" : "failed"
    },
    "retry_info" : {
      "failed" : false,
      "consumed_retries" : 0
    },
    "info" : {
      "message" : "Missing rollover_alias index setting [index=app_logs-000002]"
    },
    "enabled" : false
  },
  "total_managed_indices" : 1
}

When I do GET app_logs-000002/_settings I get:

{
  "app_logs-000002" : {
    "settings" : {
      "index" : {
        "creation_date" : "1659299029428",
        "number_of_shards" : "1",
        "number_of_replicas" : "1",
        "uuid" : "V4j0gxaYTcqoQZvtd0u2zc",
        "version" : {
          "created" : "136227827"
        },
        "provided_name" : "app_logs-000002"
      }
    }
  }
}

so yes rollover alias setting is really missing there. But I would expect that this will be added automatically. When I do GET _template I get:

{
  "ism_rollover" : {
    "order" : 0,
    "index_patterns" : [
      "app_logs-*"
    ],
    "settings" : {
      "index" : {
        "opendistro" : {
          "index_state_management" : {
            "rollover_alias" : "app_logs"
          }
        }
      }
    },
    "mappings" : { },
    "aliases" : { }
  }
}

so rollover_alias is there in template. Why this is not used in a new index from template? Thanks!

Answer 1

I experienced a similar problem. The issue was that the indices needed to be created after the ism policy and template . I'm not sure if you managed to find a solution but perhaps for those future users this could prove useful.

Some docs:

• Very useful sample on setting up a rolling index strategy: https://opensearch.org/docs/latest/im-plugin/ism/policies/#sample-policy-with-ism-template-for-auto-rollover
• Official AWS docs on the same topic with some examples: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ism.html .
• A great writeup on common errors experienced when implementing a rolling index ISM policy: https://aws.amazon.com/premiumsupport/knowledge-center/opensearch-failed-rollover-index/

In your case it appears that the policy was not correctly applying to your indices which is likely a result of you creating your indices before the policy and template were created. If you want to add a policy to an index see the step 6 of Create an ISM policy in the linked AWS docs above:

POST _plugins/_ism/add/my-index
{
  "policy_id": "my-policy-id"
}

Here is how I went about solving this problem using a policy and template:

1. Implement an ISM policy (as you did above)
2. Create an ISM template

PUT _template/ism_rollover_app
{
   "index_patterns": "app_logs-*",
    "settings": {
    "index": {
      "opendistro.index_state_management.rollover_alias": "app_logs"
    }
  }
}

3. Create an initial index called app_logs-00001 (or some variant that matches the regex ^.*-\d+$ )

This should hopefully see app_logs-00001 be created from the ism_rollover_app template and have the app_logs index associated with it. This should subsequently fix this missing alias issue.