对于Lucene（PyLucene）中的每个文档，获得TFIDF得分最高的N个术语

Question

I am currently using PyLucene but since there is no documentation for it, I guess a solution in Java for Lucene will also do (but if anyone has one in Python it would be even better).

I am working with scientific publications and for now, I retrieve the keywords of those. However, for some documents there are simply no keywords. An alternative to this would be to get N words (5-8) with the highest TFIDF scores.

I am not sure how to do it, and also when . By when, I mean : Do I have to tell Lucene at the stage of indexing to compute these values, of it is possible to do it when searching the index.

What I would like to have for each query would be something like this :

Query Ranking

Document1, top 5 TFIDF terms, Lucene score (default TFIDF)
Document2,     "       "    ,   "         "
... 

What would also be possible is to first retrieve the ranking for the query, and then compute the top 5 TFIDF terms for each of these documents.

Does anyone have an idea how shall I do this ?

Answer 1

If a field is indexed , document frequencies can be retrieved with getTerms . If a field has stored term vectors , term frequencies can be retrieved with getTermVector .

I also suggest looking at MoreLikeThis , which uses tf*idf to create a query similar to the document, from which you can extract the terms.

And if you'd like a more pythonic interface, that was my motivation for lupyne :

from lupyne import engine
searcher = engine.IndexSearcher(<filepath>)
df = dict(searcher.terms(<field>, counts=True))
tf = dict(searcher.termvector(<docnum>, <field>, counts=True))
query = searcher.morelikethis(<docnum>, <field>)

Answer 2

After digging a bit in the mailing list, I ended up having what I was looking for.

Here is the method I came up with :

def getTopTFIDFTerms(docID, reader):
    termVector = reader.getTermVector(docID, "contents");
    termsEnumvar = termVector.iterator(None)
    termsref = BytesRefIterator.cast_(termsEnumvar)
    tc_dict = {}                     # Counts of each term
    dc_dict = {}                     # Number of docs associated with each term
    tfidf_dict = {}                  # TF-IDF values of each term in the doc
    N_terms = 0
    try:
        while (termsref.next()):
            termval = TermsEnum.cast_(termsref)
            fg = termval.term().utf8ToString()       # Term in unicode
            tc = termval.totalTermFreq()             # Term count in the doc

            # Number of docs having this term in the index
            dc = reader.docFreq(Term("contents", termval.term())) 
            N_terms = N_terms + 1 
            tc_dict[fg]=tc
            dc_dict[fg]=dc
    except:
        print 'error in term_dict'

    # Compute TF-IDF for each term
    for term in tc_dict:
        tf = tc_dict[term] / N_terms
        idf = 1 + math.log(N_DOCS_INDEX/(dc_dict[term]+1)) 
        tfidf_dict[term] = tf*idf

    # Here I get a representation of the sorted dictionary
    sorted_x = sorted(tfidf_dict.items(), key=operator.itemgetter(1), reverse=True)

    # Get the top 5 
    top5 = [i[0] for i in sorted_x[:5]] # replace 5 by TOP N

I am not sure why I have to cast the termsEnum as a BytesRefIterator , I got this from a thread in the mailing list which can be found here

Hope this will help :)