Apache Lucene多个分词器

Question

I'm new to Lucene and have been trying to figure out a way to get an analyzer working.

I'd like my search string to be first split by a whitespace tokenizer, run through a keywordrepeatfilter, and then the non-keywords to be split by a standard analyzer.

Ex. "This-is some text" -> "this" "is" "this-is" "some" "text"

The WhitespaceAnalyzer alone wasn't working for what I wanted, so I started to try this, is there a way I can do this or should I try something different?

Answer 1

You can only define one Tokenizer for an Analyzer. After the Tokenizer, further modifications to the TokenStream are made using TokenFilter s. WordDelimiterFilter might be what you are looking for.

Answer 2

You can use WordDelimiterFilter to achieve what you want.

This is from lucene docs

WordDelimiterFilter Splits words into subwords and performs optional transformations on subword groups. Words are split into subwords with the following rules: - split on intra-word delimiters (by default, all non alpha-numeric characters).

The WordDelimiterFilter has PRESERVE_ORIGINAL property which willl split the "This-is some text" into following tokens

this, is,this-is,text

Read more here : https://lucene.apache.org/core/4_4_0/analyzers-common/org/apache/lucene/analysis/miscellaneous/WordDelimiterFilter.html