Hibernate 搜索模糊2个以上

Question

I have a Java backend with hibernate, lucene and hibernate-search. Now I want to do a fuzzy query, BUT instead of 0, 1, or 2, I want to allow more "differences" between the query and the expected result (to compensate for example misspelling in long words). Is there any way to achieve this? The maximum of allowed differences will later be calculated by the length of the query.
What I want this for, is an autocomplete search with correction of wrong letters. This autocomplete should only search for missing characters BEHIND the given query, not in front of it. If characters in front of the query compared to the entry are missing, they should be counted as difference.

Examples: Maximum allowed different characters in this example is 2. fooo should match

fooo       (no difference)
fooobar    (only characters added -> autocomplete)
fouubar    (characters added and misspelled -> autocomplete and spelling correction)

fooo should NOT match

barfooo    (we only allow additional characters behind the query, but this example is less important)
fuuu       (more than 2 differences)

This is my current code for the SQL query:

FullTextEntityManager fullTextEntityManager = this.sqlService.getFullTextEntityManager();
QueryBuilder queryBuilder = fullTextEntityManager.getSearchFactory().buildQueryBuilder().forEntity(MY_CLASS.class).overridesForField("name", "foo").get();
Query query = queryBuilder.keyword().fuzzy().withEditDistanceUpTo(2).onField("name").matching("QUERY_TO_MATCH").createQuery();
FullTextQuery fullTextQuery = fullTextEntityManager.createFullTextQuery(query, MY_CLASS.class);
List<MY_CLASS> results = fullTextQuery.getResultList();

Notes:
1. I use org.apache.lucene.analysis.ngram.EdgeNGramFilterFactory for indexing, but that should not make any change.
2. This is using a custom framework, which is not open source. You can just ignore the sqlService , it only provides the FullTextEntityManager and handles all things around hibernate, which do not require custom code each time.
3. This code does already work, but only with withEditDistanceUpTo(2) , which means maximum 2 "differences" between QUERY_TO_MATCH and the matching entry in the database or index. Missing characters also count as differences.
4. withEditDistanceUpTo(2) does not accept values greater than 2.

Does anyone have any ideas to achieve that?

Answer 1

I am not aware of any solution where you would specify an exact number of changes that are allowed.

That approach has serious drawbacks, anyway: what does it mean to match "foo" with up to 3 changes? Just match anything? As you can see, a solution that works with varying term lengths might be better.

One solution is to index n-grams. I'm not talking about edge-ngrams, like you already do, but actual ngrams extracted from the whole term, not just the edges. So when indexing 2-grams of foooo , you would index:

• fo
• oo (occurring multiple times)

And when querying, the term fouuu would be transformed to:

• fo
• ou
• uu

... and it would match the indexed document, since they have at least one term in common ( fo ).

Obviously there are some drawbacks. With 2-grams, the term fuuuu wouldn't match foooo , but the term barfooo would, because they have a 2-gram in common. So you would get false positives. The longer the grams, the less likely you are to get false positives, but the less fuzzy your search will be.

You can make these false positives go away by relying on scoring and on a sort by score to place the best matches first in the result list. For example, you could configure the ngram filter to preserve the original term, so that fooo will be transformed to [ fooo , fo , oo ] instead of just [ fo , oo ], and thus an exact search of fooo will have a better score for a document containing fooo than for a document containing barfooo (since there are more matches). You could also set up multiple separate fields: one without ngrams, one with 3-grams, one with 2-grams, and build a boolean query with on should clause per field: the more clauses are matched, the higher the score will be, and the higher you will find the document in the hits.

Also, I'd argue that fooo and similar are really artificial examples and you're unlikely to have these terms in a real-world dataset; you should try whatever solution you come up with against a real dataset and see if it works well enough. If you want fuzzy search, you'll have to accept some false positives: the question is not whether they exist, but whether they are rare enough that users can still easily find what they are looking for.

In order to use ngrams, apply the n-gram filter using org.apache.lucene.analysis.ngram.NGramFilterFactory . Apply it both when indexing and when querying. Use the parameters minGramSize / maxGramSize to configure the size of ngrams, and keepShortTerm ( true / false ) to control whether to preserve the original term or not.

You may keep the edge-ngram filter or not; see if it improves the relevance of your results? I suspect it may improve the relevance slightly if you use keepShortTerm = true . In any case, make sure to apply the edge-ngram filter before the ngram filter.

Answer 2

Ok, my friend and I found a solution. We found a question in the changelog of lucene which asks for the same feature, and we implemented a solution : There is a SlowFuzzyQuery in a sandbox version of lucene. It is slower (obviously) but supports an editDistance greater than 2.