spring-boot-starter-data-elasticsearch causing unknown setting analyzer/search_analyzer and index.settings.analysis.analyzer.autocomplete

Question

Context: I want to use Spring-Data to connect to ElasticSearch and retrieve data searching for at least two letters in any position of the word.

Examples:

Loaded these two documents:

[
    {
        "conta": "1234",
        "sobrenome": "Carvalho",
        "nome": "Demetrio"
    },
    {
        "conta": "5678",
        "sobrenome": "Carv",
        "nome": "Deme"
    }   
]

And searching by either "de" or "me" "rio" will bring all two and searching for "demet" brings only the first one.

I can do it succesfully by creating this index in version 7.6.2. (PUT /correntistas)

{
    "settings": {
        "index.max_ngram_diff": 10,
        "analysis": {
            "filter": {
                "autocomplete_filter": {
                    "type": "ngram",
                    "min_gram": 2,
                    "max_gram": 8
                }
            },
            "analyzer": {
                "autocomplete": {
                    "type": "custom",
                    "tokenizer": "standard",
                    "filter": [
                        "lowercase",
                        "autocomplete_filter"
                    ]
                }
            }
        }
    },
    "mappings": {
        "properties": {
            "conta": {
                "type": "text"
            },
            "sobrenome": {
                "type": "text"
            },
            "nome": {
                "type": "text",
                "analyzer": "autocomplete",
                "search_analyzer": "standard"
            }
        }
    }
}

And searching successfuly from Postman with GET correntistas/_doc/_search

{
    "query" :{
        "match" :{
            "nome" :"De"
        }
    }
}

So, no issues with own Elasticsearch at all.

But, the issue is reading searching or creating the index from Spring Data. Well, create the index straigh from Spring isn't something we will use in Production but it is worth for Developers. Nevertheless, search in ElastiSearch from Spring is a must in my case.

The complete logs is

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.2.6.RELEASE)

2020-05-11 17:53:35.050  INFO 13020 --- [           main] com.poc.search.SearchApplication         : Starting SearchApplication on SPANOT149 with PID 13020 (C:\WSs\elasticsearch\search\target\classes started by Cast in C:\WSs\elasticsearch\search)
2020-05-11 17:53:35.054  INFO 13020 --- [           main] com.poc.search.SearchApplication         : No active profile set, falling back to default profiles: default
2020-05-11 17:53:35.624  INFO 13020 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data Elasticsearch repositories in DEFAULT mode.
2020-05-11 17:53:35.685  INFO 13020 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 57ms. Found 1 Elasticsearch repository interfaces.
2020-05-11 17:53:35.844  INFO 13020 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data Reactive Elasticsearch repositories in DEFAULT mode.
2020-05-11 17:53:35.860  INFO 13020 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 15ms. Found 0 Reactive Elasticsearch repository interfaces.
2020-05-11 17:53:37.375  INFO 13020 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
2020-05-11 17:53:37.386  INFO 13020 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2020-05-11 17:53:37.387  INFO 13020 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.33]
2020-05-11 17:53:37.531  INFO 13020 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2020-05-11 17:53:37.531  INFO 13020 --- [           main] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 2379 ms
2020-05-11 17:53:38.646  INFO 13020 --- [           main] o.elasticsearch.plugins.PluginsService   : no modules loaded
2020-05-11 17:53:38.647  INFO 13020 --- [           main] o.elasticsearch.plugins.PluginsService   : loaded plugin [org.elasticsearch.index.reindex.ReindexPlugin]
2020-05-11 17:53:38.647  INFO 13020 --- [           main] o.elasticsearch.plugins.PluginsService   : loaded plugin [org.elasticsearch.join.ParentJoinPlugin]
2020-05-11 17:53:38.647  INFO 13020 --- [           main] o.elasticsearch.plugins.PluginsService   : loaded plugin [org.elasticsearch.percolator.PercolatorPlugin]
2020-05-11 17:53:38.647  INFO 13020 --- [           main] o.elasticsearch.plugins.PluginsService   : loaded plugin [org.elasticsearch.script.mustache.MustachePlugin]
2020-05-11 17:53:38.647  INFO 13020 --- [           main] o.elasticsearch.plugins.PluginsService   : loaded plugin [org.elasticsearch.transport.Netty4Plugin]
2020-05-11 17:53:41.648  INFO 13020 --- [           main] o.s.d.e.c.TransportClientFactoryBean     : Adding transport node : 192.168.99.100:9300
2020-05-11 17:53:44.203  WARN 13020 --- [           main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'correntistaController': Unsatisfied dependency expressed through field 'userService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'correntistaService': Unsatisfied dependency expressed through field 'correntistaRepository'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'correntistaRepository': Invocation of init method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.data.elasticsearch.repository.support.SimpleElasticsearchRepository]: Constructor threw exception; nested exception is java.lang.IllegalArgumentException: unknown setting [index.mappings.properties.conta.type] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
2020-05-11 17:53:45.219  INFO 13020 --- [           main] o.apache.catalina.core.StandardService   : Stopping service [Tomcat]
2020-05-11 17:53:45.228  INFO 13020 --- [           main] ConditionEvaluationReportLoggingListener : 

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2020-05-11 17:53:45.239 ERROR 13020 --- [           main] o.s.boot.SpringApplication               : Application run failed

org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'correntistaController': Unsatisfied dependency expressed through field 'userService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'correntistaService': Unsatisfied dependency expressed through field 'correntistaRepository'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'correntistaRepository': Invocation of init method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.data.elasticsearch.repository.support.SimpleElasticsearchRepository]: Constructor threw exception; nested exception is java.lang.IllegalArgumentException: unknown setting [index.mappings.properties.conta.type] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:643) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]


...


    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) ~[netty-transport-4.1.48.Final.jar:4.1.48.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) ~[na:na]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.48.Final.jar:4.1.48.Final]
    at java.lang.Thread.run(Thread.java:830) ~[na:na]
    Suppressed: java.lang.IllegalArgumentException: unknown setting [index.mappings.properties.nome.analyzer] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        ... 59 common frames omitted
    Suppressed: java.lang.IllegalArgumentException: unknown setting [index.mappings.properties.nome.search_analyzer] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        ... 59 common frames omitted
    Suppressed: java.lang.IllegalArgumentException: unknown setting [index.mappings.properties.nome.type] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        ... 59 common frames omitted
    Suppressed: java.lang.IllegalArgumentException: unknown setting [index.mappings.properties.sobrenome.type] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        ... 59 common frames omitted
    Suppressed: java.lang.IllegalArgumentException: unknown setting [index.settings.analysis.analyzer.autocomplete.filter] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        ... 59 common frames omitted
    Suppressed: java.lang.IllegalArgumentException: unknown setting [index.settings.analysis.analyzer.autocomplete.tokenizer] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        ... 59 common frames omitted
    Suppressed: java.lang.IllegalArgumentException: unknown setting [index.settings.analysis.analyzer.autocomplete.type] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        ... 59 common frames omitted
    Suppressed: java.lang.IllegalArgumentException: unknown setting [index.settings.analysis.filter.autocomplete_filter.max_gram] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        ... 59 common frames omitted
    Suppressed: java.lang.IllegalArgumentException: unknown setting [index.settings.analysis.filter.autocomplete_filter.min_gram] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        ... 59 common frames omitted
    Suppressed: java.lang.IllegalArgumentException: unknown setting [index.settings.analysis.filter.autocomplete_filter.type] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        ... 59 common frames omitted
    Suppressed: java.lang.IllegalArgumentException: unknown setting [index.settings.index.max_ngram_diff] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        ... 59 common frames omitted

The whole project can be found in my GitHub repository under search folder.

I believe the relevant files are

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.2.6.RELEASE</version>
        <relativePath /> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.poc</groupId>
    <artifactId>search</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>search</name>
    <description>Demo project for search-as-user-type cases</description>

    <properties>
        <java.version>11</java.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
            <!-- version>2.2.7.RELEASE</version-->
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <!-- <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> 
            <optional>true</optional> </dependency> -->

        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <version>1.18.12</version>
            <scope>provided</scope>
        </dependency>


        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
            <exclusions>
                <exclusion>
                    <groupId>org.junit.vintage</groupId>
                    <artifactId>junit-vintage-engine</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

model

import lombok.Getter;
import lombok.Setter;
import org.springframework.data.annotation.Id;
import org.springframework.data.elasticsearch.annotations.Document;
import org.springframework.data.elasticsearch.annotations.Field;
import org.springframework.data.elasticsearch.annotations.FieldType;
import org.springframework.data.elasticsearch.annotations.Setting;

@Document(indexName = "correntistas")
@Setting(settingPath = "data/es-config/elastic-setting.json")
@Getter
@Setter
public class Correntista {
    @Id
    private String id;
    private String conta;
    private String sobrenome;

    @Field(type = FieldType.Text, analyzer = "autocomplete_index", searchAnalyzer = "autocomplete_search")
    private String nome;
}

elastic-setting.json (exact same index put from Postman)

{
    "settings": {
        "index.max_ngram_diff": 10,
        "analysis": {
            "filter": {
                "autocomplete_filter": {
                    "type": "ngram",
                    "min_gram": 2,
                    "max_gram": 8
                }
            },
            "analyzer": {
                "autocomplete": {
                    "type": "custom",
                    "tokenizer": "standard",
                    "filter": [
                        "lowercase",
                        "autocomplete_filter"
                    ]
                }
            }
        }
    },
    "mappings": {
        "properties": {
            "conta": {
                "type": "text"
            },
            "sobrenome": {
                "type": "text"
            },
            "nome": {
                "type": "text",
                "analyzer": "autocomplete",
                "search_analyzer": "standard"
            }
        }
    }
}

repository

import org.elasticsearch.index.query.QueryBuilder;
import org.springframework.data.elasticsearch.repository.ElasticsearchRepository;
import org.springframework.stereotype.Repository;

import com.poc.search.model.Correntista;

import java.util.List;

@Repository
public interface CorrentistaRepository extends ElasticsearchRepository<Correntista, Long> {
    List<Correntista> findAll();

    List<Correntista> search(QueryBuilder query);
}

I am aware that there is some incompatibility between Elastic 6 and 7 and that Spring-data still doesn't have 7 dependency as far as I can see. Btw, I can't downgrade ElasticSearch from 7 to 6 and I do have do code a Microservice using Spring for searching as user type from any position in the word. I believe it is a problem or issue to be answer mainly from Spring-data-elasticsearch supporters but maybe there ia another alternative someone can propose coming from Elasticsearch users. Please, don't suggest me to use neither * nor regular expression as an alternative unless you are 100% it is going to take advantage of ElasticSearch engine (I am pretty sure it is a bad performance practice use * during search on Elastic)

Answer 1

Where do you have the settings file stored?

@Setting(settingPath = "data/es-config/elastic-setting.json")

would mean that the data directory is in the same level as the entity class. If you have it in src/main/resources/data/es-config change the value:

@Setting(settingPath = "/data/es-config/elastic-setting.json")

Edit: To use Spring Data Elasticsearchwith Elasticsearch 7, you need Spring Data Elasticsearch 4.0, released on May 12th