索引和查询时多个令牌过滤器的Apache Solr性能问题

Question

I have to convert Number numbers from one language to other in Apache Solr 6.6.2. For that I have found pattern replacement filter could do this job. I have added a new field in Solr schema with following filters

<fieldType name="text_use" class="solr.TextField">
    <analyzer type="index">
      <tokenizer class="solr.StandardTokenizerFactory"/>
    <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/>
    <filter class="solr.LowerCaseFilterFactory"/>

    <filter class="solr.PatternReplaceFilterFactory" pattern="0" replacement="۰"/>
    <filter class="solr.PatternReplaceFilterFactory" pattern="1" replacement="۱"/>
    <filter class="solr.PatternReplaceFilterFactory" pattern="2" replacement="۲"/>
    <filter class="solr.PatternReplaceFilterFactory" pattern="3" replacement="۳"/>
    <filter class="solr.PatternReplaceFilterFactory" pattern="4" replacement="۴"/>
    <filter class="solr.PatternReplaceFilterFactory" pattern="5" replacement="۵"/>
    <filter class="solr.PatternReplaceFilterFactory" pattern="6" replacement="۶"/>
    <filter class="solr.PatternReplaceFilterFactory" pattern="7" replacement="۷"/>
    <filter class="solr.PatternReplaceFilterFactory" pattern="8" replacement="۸"/>
    <filter class="solr.PatternReplaceFilterFactory" pattern="9" replacement="۹"/>
    </analyzer>

    <analyzer type="query">
      <tokenizer class="solr.StandardTokenizerFactory"/>
    <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/>
    <filter class="solr.SynonymGraphFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/>
    <filter class="solr.LowerCaseFilterFactory"/>

    <filter class="solr.PatternReplaceFilterFactory" pattern="0" replacement="۰"/>
    <filter class="solr.PatternReplaceFilterFactory" pattern="1" replacement="۱"/>
    <filter class="solr.PatternReplaceFilterFactory" pattern="2" replacement="۲"/>
    <filter class="solr.PatternReplaceFilterFactory" pattern="3" replacement="۳"/>
    <filter class="solr.PatternReplaceFilterFactory" pattern="4" replacement="۴"/>
    <filter class="solr.PatternReplaceFilterFactory" pattern="5" replacement="۵"/>
    <filter class="solr.PatternReplaceFilterFactory" pattern="6" replacement="۶"/>
    <filter class="solr.PatternReplaceFilterFactory" pattern="7" replacement="۷"/>
    <filter class="solr.PatternReplaceFilterFactory" pattern="8" replacement="۸"/>
    <filter class="solr.PatternReplaceFilterFactory" pattern="9" replacement="۹"/>
    </analyzer>
</fieldType>

Applying a lot of filters at query and index time is good ? Is there an any performance issue in system due to large number of filter ? Finally, Is it possible to write one filter with regex pattern ? If yes then what will be that ?

Answer 1

Performance - try it both with and without. Since indexing is usually done without a very hard performance requirement, it's usually not a very large issue if it takes a few milliseconds extra. For queries it's just the query text that's being processed, and that is far less content than the documents themselves.

I don't think there as an easier way to do what you want with the patternreplacementfilter, since you're looking for a specific replacement for each digit.

Writing your own filter would probably be the easiest way - an urdu numeric conversion filter, which probably would be useful for more people as well (so upload it to a github repo). In a separate filter you can perform all the replacements in a single go, and you can do it without regex support (although the performance difference might not be much, it should at least be faster than invoking the regex engine ten times - but again, test it yourself).