Solr（查询分析器）中NGramFilterFactory的Tokenize结果

Question

I'm using the NGramFilterFactory for indexing and querying.

So if I'm searching for "overflow" it creates an query like this:

mySearchField:"ov ve ... erflow overflo verflow overflow"

But if I misspell "overflow", ie "owerflow" there are no matches, because the quotes around the query:

mySearchField:"ow we ... erflow owerflo werflow owerflow"

Is it possible to tokenize the result of the NGramFilteFactory, that it'll creates an query like this:

mySearchField:"ow"
mySearchField:"we"
mySearchField:"erflow"
mySearchField:"owerflo"
mySearchField:"werflow"
mySearchField:"owerflow"

In this case solr would also find results, because the token "erflow" exists.

Answer 1

You don't need to tokenize your query like you wrote. Check if in your schema.xml you have the NGramFilterFactory applied at both index time and query time. Then, the query parser you're using makes the difference. With LuceneQParser you'd get the result you're looking for, but not with DisMax and eDisMax .

I checked the query mySearchField:owerflow with eDisMax and debugQuery=on :

<str name="querystring">text:owerflow</str>
<str name="parsedquery">
+((text:o text:w text:e text:r text:f text:l text:o text:w text:ow text:we text:er text:rf text:fl text:lo text:ow text:owe text:wer text:erf text:rfl text:flo text:low text:ower text:werf text:erfl text:rflo text:flow text:owerf text:werfl text:erflo text:rflow text:owerfl text:werflo text:erflow text:owerflo text:werflow text:owerflow)~36)
</str>

If you look at the end of the generated query you'll see ~36 where 36 is the number of n-grams generated from your query. You don't get any results because of that ~36 , but you can change it through the mm parameter, which is the minimum should match.

If you change the query to mySearchField:owerflow&mm=1 or a value lower than 25 you'll have the result you're looking for.

The difference between this answer and yours is that with EdgeNGramFilterFactory an infix query like mySearchField:werflow doesn't return any result, while it does with NGramFilterFactory .

Anyway, If you're using the NGramFilterFactory for making spelling correction, I'd strongly recommend to have a look at the SpellCheckComponent as well, made exactly for that purpose.

Answer 2

OK, I found a quick and easy way to solve the problem.

The fieldType has an optional attribute autoGeneratePhraseQueries (Default=true). If I set autoGeneratePhraseQueries to false, everything works fine.

Explanation:

fieldType used in schema.xml:

<fieldType name="edgytext" class="solr.TextField" autoGeneratePhraseQueries="false">
 <analyzer type="index">
   <tokenizer class="solr.KeywordTokenizerFactory"/>
   <filter class="solr.LowerCaseFilterFactory"/>
   <filter class="solr.EdgeNGramFilterFactory" minGramSize="1" maxGramSize="25" />
 </analyzer>
 <analyzer type="query">
   <tokenizer class="solr.WhiteSpaceTokenizerFactory"/>
   <filter class="solr.LowerCaseFilterFactory"/>
   <filter class="solr.EdgeNGramFilterFactory" minGramSize="1" maxGramSize="25" />
 </analyzer>
</fieldType>

If you are indexing the word "surprise", following tokens are in the index:

s, su, ,sur, surp, surpr, surpri, surpris, surprise

If you are search for "surpriese" (misspelled) solr creates following tokens (matching tokens are bold):

s , su , sur , surp , surpr , surpri , surprie, surpries, surpriese

The real query which will be created looks like:

mySearchField:s, mySearchField:su, mySearchField:sup .. and so on

But if you set autoGeneratePhraseQueries=true following query will be created:

mySearchField:"s su surp supr surprie surpries surpriese"

This is an phrase query and does not match the indexed terms.