如何使用语言 model 扩展 Lucene？

Question

Good evening everyone, So, i have as project "extend lucene with a language model", i tried adding 2: 3 lines to my code like this. on search.java

LMDirichletSimilarity similarity = new LMDirichletSimilarity(2000f);  
indexSearcher.setSimilarity(similarity);

and on index.java

LMDirichletSimilarity similarity = new LMDirichletSimilarity(2000f);
config.setSimilarity(similarity);

but i don't think it's that easy? maybe i should write an algorithm or something ? please if you have some answers help me thank you ^_^

Answer 1

There is a language model similarity method in LMJelinekMercerSimilarity and the implementation is:

protected float score(BasicStats stats, float freq, float docLen) {
    return stats.getBoost()
            * (float) Math.log(1 + ((1 - alpha) * freq / docLen)
                            / (alpha * ((LMStats) stats).getCollectionProbability()));
}

This method is implementation of this formula: (1-lambda) * P(w|d) + lambda * P(w|Collection) If you look at the method above and the language model formula you see there is a bit difference between them. It is because Lucene's factorizes the expression lambda * P(w|Collection) from the language model formula and create a new formula: lambda * P(w|Collection) * ( ( (1-lambda) * P(w|d) / lambda * P(w|Collection) ) +1 ) then it removes the lambda * P(w|Collection) because of ranking (It doesn't affect ranking) and just calculate ( ( (1-lambda) * P(w|d) / lambda * P(w|Collection) ) +1 ) . you can see it is similar the method above. but there is a little difference and that is Logarithm. In IR community they use Logarithm because that is easy to deal and easily evaluated by computers. so the final statement is: log ( ( ( (1-lambda) * P(w|d) / lambda * P(w|Collection) ) +1 ) ) The method above is a protected so you can derive that method and implement your own.