如何限制magento elasticsearch索引的字段数

Question

EDIT2: Was able to get over this error, but now have another one about mapper conflicts. Apparently there already is some other mapper and it tries to update the field (?).

I think I'll give up for now. Checked indices and they are full of fields again:

"price_749_12":{"type":"double","store":true},"price_749_7":{"type":"double","store":true},"price_750_12":{"type":"double","store":true},"price_750_7":{"type":"double","store":true},"price_751_12":{"type":"double","store":true},"price_751_7":{"type":"double","store":true},"price_752_12":{"type":"double","store":true},"price_752_7":{"type":"double","store":true},"price_753_12":{"type":"double","store":true},"price_753_7":{"type":"double","store":true},"price_754_12":{"type":"double","store":true},"price_754_7":{"type":"double","store":true},"price_755_12":{"type":"double","store":true},"price_755_7":{"type":"double","store":true},"price_756_12":{"type":"double","store":true},"price_756_7":{"type":"double","store":true},"price_759_12":{"type":"double","store":true},"price_759_7":{"type":"double","store":true},"price_760_12":{"type":"double","store":true},"price_760_7":{"type":"double","store":true},"price_761_12":{"type":"double","store":true},"price_761_7":{"type":"double","store":true},"price_763_12":{"type":"double","store":true},"price_763_7":{"type":"double","store":true},"price_764_12":{"type":"double","store":true},"price_764_7":{"type":"double","store":true},"price_765_12":{"type":"double","store":true},"price_765_7":{"type":"double","store":true},"price_766_12":{"type":"double","store":true},"price_766_7":{"type":"double","store":true},"price_767_12":{"type":"double","store":true},"price_767_7":{"type":"double","store":true}

---

---

EDIT: Ok, as of now, I came to the index templates and such. Also got a little more knowledge about API and various options. Made a json as a payload to the PUT request, along with settings, analyzers, filters and so on.

Indeed, fields count after applying the template fell down to 684.

Main problem is an error while reindexing catalog search index:

Rejecting mapping update to [rcb2b_product_1_v2] as the final mapping would have more than 1 type: [_doc, document]"}]

It's elastic 7.13.1 and I am looking for an option to put in the right place to make it work...

---

---

I've got a problem with product index (which is in fact the only index in magento 2) being bloated to the point that it has over 34000 fields. Site has over hundred of customer groups and over 40 countries/websites. Magento indexes prices (and a few other things) for any combination of them.

Now, trying to view indexes in kibana often fails, because of "limit exceeded" error. This is problem number one, but problem number two is more generic - I'm just concerned about speed penalty for such a setup.

Thought about some ways to solve the problem. Even created static mapping with index set to false like this:

"dynamic_templates": [
        {
          "price_mapping": {
            "match": "price_*",
            "match_mapping_type": "string",
            "mapping": {
              "index": false,
              "store": true,
              "type": "double"
            }
          }
        },

but I suppose it would crash the site, not to mention that I don't even know yet how to apply it to constantly recreated indexes.

Also thought about fixing kibana queries so that they would not expand automatically [*]. No idea how to get to this either.

Last but not least, I could just set index.mapping.total_fields.limit to some absurd value and see what happens. It probably would not kill the site, but might kill kibana, if expanding queries come only from its side.

Well. Any ideas?

Answer 1

There are multiple question and I tried to answer them one by one.

So, first, you need to use index templates to apply the mapping you created already to all indices related to product indices. Also you can check using aliases for the product indices. There is an option on aliases as is_write_index . I think you need to check it, too.

Second, there is a dynamic parameter for the properties of the mapping. And you can prevent adding the all fields to mapping even.

https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic.html#dynamic-parameters

In the document, you can see that, there is a dynamic: false option there. This will ignore the fields. But this means that you can not do a search on that fields. So, for example, you have object and there are so many fields inside this object and you are not using that fields in the search. You can use following mapping for this object to prevent adding that fields into mapping.

PUT products
{
  "mappings": {
    "dynamic": false, 
    "properties": {
      "name": {
        "type": "text"
      },
      "extras": {
        "dynamic": false, 
        "properties": {}
      }
    }
  }
}

In this example, all the fields inside the extras object will be ignored to adding into mapping.

Increasing the index.mapping.total_fields.limit configuration is a solution but I think we need to keep it low if there is a limit somewhere. The limits means that Elasticsearch using much resource according to this limit when we increase it. Instead of, you can try to other solutions first. You can try to increase that limit if you don't find a solution in short.