使用logstash在弹性搜索中将两个索引组合成第三个索引

Question

I have two index

1. employee_data {"code":1, "name":xyz, "city":"Mumbai" }
2. transaction_data {"code":1, "Month":June", payment:78000 }

I want third index like this 3)join_index

{"code":1, "name":xyz, "city":"Mumbai", "Month":June", payment:78000 } How it's possible??

i am trying in logstash

input {
  elasticsearch {
    hosts => "localost"
    index => "employees_data,transaction_data"
   
     query => '{ "query": { "match": { "code": 1} } }'
    scroll => "5m"
    docinfo => true
  }
}
output {

elasticsearch { hosts => ["localhost"]

index => "join1"
   }

}

Answer 1

You can use elasticsearch input on employees_data

In your filters, use the elasticsearch filter on transaction_data

input {
  elasticsearch {
    hosts => "localost"
    index => "employees_data"
   
     query => '{ "query": { "match_all": { } } }'
     sort => "code:desc"

    scroll => "5m"
    docinfo => true
  }
}
filter {
    elasticsearch {
              hosts => "localhost"
              index => "transaction_data"
              query => "(code:\"%{[code]}\"
              fields => { 
                    "Month" => "Month",
                    "payment" => "payment" 
                   }
        }
}
output {
  elasticsearch { 
    hosts => ["localhost"]
    index => "join1"
   }
}

And send your new document to your third index with the elasticsearch output

You'll have 3 elastic search connection and the result can be a little slow. But it works.

Answer 2

You don't need Logstash to do this, Elasticsearch itself supports that by leveraging the enrich processor .

First, you need to create an enrich policy (use the smallest index, let's say it's employees_data ):

PUT /_enrich/policy/employee-policy
{
  "match": {
    "indices": "employees_data",
    "match_field": "code",
    "enrich_fields": ["name", "city"]
  }
}

Then you can execute that policy in order to create an enrichment index

POST /_enrich/policy/employee-policy/_execute

When the enrichment index has been created and populated, the next step requires you to create an ingest pipeline that uses the above enrich policy/index:

PUT /_ingest/pipeline/employee_lookup
{
  "description" : "Enriching transactions with employee data",
  "processors" : [
    {
      "enrich" : {
        "policy_name": "employee-policy",
        "field" : "code",
        "target_field": "tmp",
        "max_matches": "1"
      }
    },
    {
      "script": {
        "if": "ctx.tmp != null",
        "source": "ctx.putAll(ctx.tmp); ctx.remove('tmp');"
      }
    }
  ]
}

Finally, you're now ready to create your target index with the joined data. Simply leverage the _reindex API combined with the ingest pipeline we've just created:

POST _reindex
{
  "source": {
    "index": "transaction_data"
  },
  "dest": {
    "index": "join1",
    "pipeline": "employee_lookup"
  }
}

After running this, the join1 index will contain exactly what you need, for instance:

  {
    "_index" : "join1",
    "_type" : "_doc",
    "_id" : "0uA8dXMBU9tMsBeoajlw",
    "_score" : 1.0,
    "_source" : {
      "code":1, 
      "name": "xyz", 
      "city": "Mumbai", 
      "Month": "June", 
      "payment": 78000 
    }
  }

Answer 3

As long as I know, this can not be happened just using elasticsearch APIs. To handle this, you need to set a unique ID for documents that are relevant. For example, the code that you mentioned in your question can be a good ID for documents. So you can reindex the first index to the third one and use UPDATE API to update them by reading documents from the second index and update them by their IDs into the third index. I hope I could help.