为什么在Lucene的同一文档中得到不同的术语？

Question

I have an sentence-based index where every document stores the content and a field for every token. The content field is named content and stores the whole sentence as String, the token fields are named token0 , token1 , ... tonken_n-1 containing each token as a String.

When I used the code sample in the Apache Migration guide to get all unique terms for each field in the example sample "This index is sentence-based.",

for(String field : fields) {
   Terms terms = fields.terms(field);
   TermsEnum termsEnum = terms.iterator(null);
   BytesRef text;
   while((text = termsEnum.next()) != null) {
     System.out.println("field=" + field + "; text=" + text.utf8ToString());
   }
}

sentence-based is recognized as term in the field token3 , but in the content field only sentence and based based is recognized. It seems like fields.terms(field) uses a different Analyzer for each field.

I have no clue why I get different terms when applying fields.terms("content") . I want to get sentence-based as a term out of the content field instead of sentence and based .

I hope there is an explanation for this phenomenon.

Answer 1

It sounds like the content field is being analyzed and the tokenN fields aren't. Analysis happens at index time, so what you need to do is revisit your indexing code and find out why content and token fields are being analyzed differently.

If you want the content field to be analyzed, just differently from what you have now, then okay. Do that! SimpleAnalyzer may be a good place to start.