Elasticsearch聚合，将文档字段与多个存储桶的存储桶键进行比较

Question

I have an index of orders. Each document in the index contains a date that the order was completed. I am trying to build an aggregation that gives me the historical work in progress (wip) for a date histogram aggregation. The wip is calculated by comparing the completed date with each date in the date histogram. If the completed date is > current bucket date then the order is considered in progress and should be included in the bucket.

From my research the best I can determine is that a date_histogram using a value script would give me the results I need. However I can't figure out how to structure my script.

Currently my query looks like this:

{
    "query": {
        "match_all": {}
    },
    "aggs": {
        "wip": {
            "date_histogram": {
                "field": "com_ord_created_ddate",
                "script": "doc['com_ord_completed_ddate'] > _value",
                "interval": "day",
                "format": "yyyy-MM-dd"
            }
        }
    }
}

This query returns the following exception

{
  "error": {
    "root_cause": [
      {
        "type": "script_exception",
        "reason": "failed to run inline script [doc['com_ord_completed_ddate'] > _value] using lang [groovy]"
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "orders",
        "node": "eYYqpuNSQ0KOt04JEztTDg",
        "reason": {
          "type": "script_exception",
          "reason": "failed to run inline script [doc['com_ord_completed_ddate'] > _value] using lang [groovy]",
          "caused_by": {
            "type": "groovy_runtime_exception",
            "reason": "Cannot compare org.elasticsearch.index.fielddata.ScriptDocValues$Longs with value '[1445864743000]' and java.lang.Long with value '1,445,618,646,000'"
          }
        }
      }
    ]
  },
  "status": 500
}

I know script is written poorly. I have not been able to find documentation that clearly outlines available variables inside the script scope. I got _value from a qbox tutorial here https://qbox.io/blog/elasticsearch-scripting-aggregations But not enough is said about what _value is and what other variables are available for operating on.

Can anyone point me to clear documentation on aggregation inline value scripting or can they help me by providing a script that would get the results that I need?

UPDATE I was able to get the first part of my question using this:

{
    "query": {
        "match_all": {}
    },
    "aggs": {
        "wip": {
            "date_histogram": {
                "field": "com_ord_created_ddate",
                "script": "if(_value < doc['com_ord_completed_ddate'].value) {_value} else {0}",
                "interval": "day",
                "format": "yyyy-MM-dd"
            }
        }
    }
}

However the script is limited to comparing documents already aggregated to the bucket. I need to compare all documents in the result for every bucket. Any thoughts?

Answer 1

Please try the following query

{
"query": {
    "match_all": {}
},
"aggs": {
    "wip": {
        "date_histogram": {
            "field": "com_ord_created_ddate",
            "script": "doc['com_ord_completed_ddate'].value > _value",
            "interval": "day",
            "format": "yyyy-MM-dd"
        }
    }
}

Note the .value after doc['com_ord_completed_ddate']

Documentation for Script aggregation

Answer 2

I was really close to getting what I needed with a scripted_metric aggregation that looked like this.

"aggs": {
    "wip": {
        "scripted_metric": {
            "init_script": "_agg['created_dates'] = []; _agg['documents'] = []",
            "map_script": "_agg.created_dates.add(doc['com_ord_created_ddate'].value); document = [:]; document.created_date = doc['com_ord_created_ddate']; document.completed_date = doc['com_ord_completed_ddate']; _agg.documents.add(document); return _agg;",
            "combine_script": "_agg.created_dates.unique()",
            "reduce_script": "results = []; wip = [:];for (agg in _aggs) { for (d in agg.created_dates) {wip.key = d; wip.doc_count = 0; for(o in agg.documents) { if (d < o.completed_date && d >= o.created_date) { wip.doc_count++ } }; results.add(wip); }; }; return results;"
        }
    }
}

Unfortunately the script is poorly optimized and the query was taking 10 - 15 seconds, which for ES is an eternity and did not meet performance requirements for the project.

In the end I was able to get the results I needed by making multiple ES queries. I first queried the above mentioned date_histogram aggregation without the script parameter. Then serverside I looped over each date that was returned in the aggregation and performed another query to get my historical wip for that specific day. Its not pretty but surprisingly it was quicker than the scripted_metric. Total load time for the end user is about 2-3 seconds which is an acceptable time frame in my use case.