与Lucene的余弦相似度仅适用于匹配的文档

Question

Lucene is a inverse indexing system, as far as I understand, its power lies in the fact that it will compare a query only with documents that at least match a token.

Compared to the naive approach where the query is compared to every document, (even those that don't mention any token that is present in the query) this is a great benefit.

For example if I have the indexed documents:

D1: "Hello world said the guy"
D2: "Hello, what a beautiful world"
D3: "random text"

As I see it , the search for query: " Hello world ", will only look into the indexed documents D1 and D2 and skips on D3, which saves time.

Is this correct?

Now, I'm trying to calculate the cosine similarity between documents. The input query will be a document and the output should be the cosine score. Which is a number between 0 and 1.

I've already read some approaches that calculate the cosine similarity, but they all do this by comparing the term vector for every document. For example this blog mentioned the following:

If you really need cosine similarity between documents, you have to enable term vectors for the source fields, and use them to calculate the angle. The problem is that this does not scale well, you would need to calculate angles with virtually all other documents .

and this SO answers seems to say the same:

3. iterate over all doc ids , 0 to maxDoc();

Isn't there a way to only calculate the cosine similarity for documents that match the query and let this return as score for the document?

As a side note, I did read that the TFIDFSimilarity comes close, I believe the VSM part is exactly what I need, however this part seems to have disappeared in the Lucene Practical Scoring Function. I'm not sure how I can "transform" this Similarity class to end up with only the pure cosine similarity as result.

So a recap of my question:

1. Is my perception of how the inverse indexes save time correct?

2. Is there way to only calculate cosine similarity for documents that actually match one of the tokens, instead of for all the documents?

3. Can I use/transform the TFIDFSimilarity class to end up with the pure cosine similarity?

Answer 1

2. It pretty much depends on how you formulate your query. If you formulate a BooleanQuery you can specify which terms of the query must be in the returned document. This is done using BoolenClause.Occur.MUST .

3. You can write your own similarity by extending TFIDFSimilarity but as you may noticed the Lucene practical scoring is based on cosine similarity. In that formula, queryNorm(q) and norm(t, d) form the denominator of cosine similarity and the summation is the dot product of query vector and document vector.

hint: you can form a sample query and use explain() to see the details of scoring.