Lucene中LMJelinekMercerSimilarity方法中使用的LM公式的解释

Question

I found a return statement in score method of LMJelinekMercerSimilarity like:

protected float score(BasicStats stats, float freq, float docLen) {
    return stats.getBoost()
            * (float) Math.log(1 + ((1 - alpha) * freq / docLen)
                            / (alpha * ((LMStats) stats).getCollectionProbability()));
}

This return statement should be theoretically similar as: "(1-lambda) * P(w|d) + lambda * P(w|Collection)".

But, I cannot understand how they are related. Can anyone help.

Answer 1

The implementation of lucene is a bit different from The language Model.

The language model actually calculates the Probability of a query (in the specified context) and it uses the JM model:

(1-lambda) * P(w|d) + lambda * P(w|Collection)

But Lucene does some mathematical operations on this expression. It factorizes the expression lambda * P(w|Collection) from the language model formula and obtains:

lambda * P(w|Collection) * ( ( (1-lambda) * P(w|d) / lambda * P(w|Collection) ) +1 )

By the concept of ranking models, We know that the first factor: lambda * P(w|Collection) doesn't affect the final ranking, So lucene ignores this factor and obtains this expression:

( ( (1-lambda) * P(w|d) / lambda * P(w|Collection) ) +1 )

In Information Retrieval trend, We're desired to work with log-scale functions. Because they are easy to deal and that's the reason why lucene uses log function:

log ( ( ( (1-lambda) * P(w|d) / lambda * P(w|Collection) ) +1 ) )

Hope this explanation would help you!