还有其他方法可以针对JSON中的多个嵌套字段优化此Elasticsearch查询

Question

I am new to elasticserach. Below is the Sample data on which elastic query needs to run. I am trying to get those docs in which account_type is "credit card" and source_name is 'SOMEVALUE'

{
"took" : 0,
"timed_out" : false,
"_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
},
"hits" : {
    "total" : {
    "value" : 1,
    "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
    {
        "_index" : "bureau_data",
        "_type" : "_doc",
        "_id" : "bda57e01-c564-4cdc-bb8d-79bd2db9d2f8",
        "_score" : 1.0,
        "_source" : {
        "userid" : "bda57e01-c564-4cdc-bb8d-79bd2db9d2f8",
        "raw_derived" : {
            "gender" : "MALE",
            "firstname" : "trsqlsz",
            "middlename" : "rgj",
            "lastname" : "ggksb",
            "mobilephone" : "2125954664",
            "dob" : "1988-06-28 00:00:00",
            "applications" : [
            {
                "applicationid" : "c7fb0147-22fd-4a5e-8851-98241de6aa50",
                "createdat" : "2019-06-07 19:28:54",
                "updatedat" : "2019-06-07 19:28:55",
                "source" : "4",
                "source_name" : "EXPERIAN",
                "applicationcreditreportid" : "b67f9180-9bb6-485c-9cfc-e7ccf9a70a69",
                "accounts" : [
                {
                    "applicationcreditreportaccountid" : "c5de28c4-cac9-4390-852a-96f143cb0b62",
                    "currentbalance" : 418288,
                    "institutionid" : "021d58b4-aba5-42c9-8d39-304a78d34aea",
                    "accounttypeid" : "5",
                    "institution_name" : "HDFC BANK",
                    "account_type_name" : "Personal Loan"
                }
                ]
            }
            ]
        }
        }
    }

I have tried the below query and its working fine. I need if we have any optimized way to query the multiple nested fields

GET /my_index/_search
{
"query": {
    "bool": {
    "must": [
        {
        "nested": {
            "path": "raw_derived.applications.accounts",
            "query": {
            "bool": {
                "must": [
                {"match": {
                    "raw_derived.applications.accounts.account_type_name": "Credit Card"
                }}
                ]
            }
            }
        }
        },
        {
        "nested": {
            "path": "raw_derived.applications",
            "query": {
            "bool": {
                "must": [
                {"match": {
                    "raw_derived.applications.source_name": "CIBIL"
                }}
                ]
            }
            }
        }
        }
    ]
    }
}

}

If I will query on the multiple nested fields it will become very long Please suggest any other way to query nested fields or multiple AND

Answer 1

Well your optimizations should always start with your data model / mapping since it's mostly the cause of performance issues and not your queries.

That being said, you can avoid the nested query by flattening your data. A flattened data model would lead to one document per application and account element.

Since elasticsearch is a non-relational data store, it is completely fine to index "redundant" data. This is not a lazy appraoch but a common way to handle these type of data structures.

Sample document #1:

{
    "_index" : "bureau_data",
    "_type" : "_doc",
    "_id" : "bda57e01-c564-4cdc-bb8d-79bd2db9d2f8",
    "_score" : 1.0,
    "_source" : {
      "userid" : "bda57e01-c564-4cdc-bb8d-79bd2db9d2f8",
      "gender" : "MALE",
      "firstname" : "trsqlsz",
      "middlename" : "rgj",
      "lastname" : "ggksb",
      "mobilephone" : "2125954664",
      "dob" : "1988-06-28 00:00:00",
      "applicationid" : "c7fb0147-22fd-4a5e-8851-98241de6aa50",
      "createdat" : "2019-06-07 19:28:54",
      "updatedat" : "2019-06-07 19:28:55",
      "source" : "4",
      "source_name" : "EXPERIAN",
      "applicationcreditreportid" : "b67f9180-9bb6-485c-9cfc-e7ccf9a70a69",
      "applicationcreditreportaccountid" : "c5de28c4-cac9-4390-852a-96f143cb0b62",
      "currentbalance" : 418288,
      "institutionid" : "021d58b4-aba5-42c9-8d39-304a78d34aea",
      "accounttypeid" : "5",
      "institution_name" : "HDFC BANK",
      "account_type_name" : "Personal Loan"
    }
}

If the same user creates another account you would send the very same ("redundant") data, except for that other account element/data like so:

    {
    "_index" : "bureau_data",
    "_type" : "_doc",
    "_id" : "another, from es generated id",
    "_score" : 1.0,
    "_source" : {
      "userid" : "bda57e01-c564-4cdc-bb8d-79bd2db9d2f8",
      "gender" : "MALE",
      "firstname" : "trsqlsz",
      "middlename" : "rgj",
      "lastname" : "ggksb",
      "mobilephone" : "2125954664",
      "dob" : "1988-06-28 00:00:00",
      "applicationid" : "c7fb0147-22fd-4a5e-8851-98241de6aa50",
      "createdat" : "2019-06-07 19:28:54",
      "updatedat" : "2019-06-07 19:28:55",
      "source" : "4",
      "source_name" : "EXPERIAN",
      "applicationcreditreportid" : "b67f9180-9bb6-485c-9cfc-e7ccf9a70a69",
      "applicationcreditreportaccountid" : "the new id",
      "currentbalance" : 4711,
      "institutionid" : "foo",
      "accounttypeid" : "bar",
      "institution_name" : "foo bar",
      "account_type_name" : "foo baz"
    }
}

With that kind of data model, you can run simple queries to get your results:

    GET /my_index/_search
{
    "query": {
        "bool": {
            "must": [
            {
                "match":{
                    "account_type_name": "Credit Card"
                } 
            },
            {
                "match":{
                    "source_name": "CIBIL"
                } 
            }
            ]
        }
    }
}