Checking if field exists under an elasticsearch nested aggregation

Question

Trying to perform an ES query, I ran into a problem while trying to do a nested filtering of objects in an array. Our structure of data has changed from being:

 "_index": "events_2015-07-08",
 "_type": "my_type",
 "_source":{
    ...
    ...
    "custom_data":{
        "className:"....."
    }
 }

to:

 "_index": "events_2015-07-08",
 "_type": "my_type",
 "_source":{
    ...
    ...
    "custom_data":[   //THIS CHANGED FROM AN OBJECT TO AN ARRAY OF OBJECTS
        {
          "key":".....",
         "val":"....."
        },
        {
          "key":".....",
         "val":"....."
        }
    ]
 }

this nested filter works fine on indices that have the new data structure:

{
     "nested": { 
         "path": "custom_data",
         "filter": {
             "bool": {
                 "must": [                                
                    {
                        "term": 
                            {
                             "custom_data.key": "className"
                            }
                    }, 
                    {
                         "term": {
                             "custom_data.val": "SOME_VALUE"
                         }
                     }
                  ]
              }
          },
          "_cache": true   
      }
 }

However, it fails when going over indices that have the older data structure, so that feature cannot be added. Ideally I'd be able to find both data structures but at this point i'd settle for a "graceful failure" ie just don't return results where the structure is old.

I have tried adding an "exists" filter on the field "custom_data.key", and an "exists" within "not" on the field "custom_data.className", but I keep getting "SearchParseException[[events_2015-07-01][0]: from[-1],size[-1]: Parse Failure [Failed to parse source"

Answer 1

There is an indices filter (and query) that you can use to perform conditional filters (and queries) based on the index that it is running against.

{
  "query" : {
    "filtered" : {
      "filter" : {
        "indices" : {
          "indices" : ["old-index-1", "old-index-2"],
          "filter" : {
            "term" : {
              "className" : "SOME_VALUE"
            }
          },
          "no_match_filter" : {
            "nested" : { ... }
          }
        }
      }
    }
  }
}

Using this, you should be able to transition off of the old mapping and onto the new mapping.