After applying gensim LDA topic modeling, how to get documents with highest probability for each topic and save them in a csv file?

Question

I have used gensim LDA Topic Modeling to get associated topics from a corpus. Now I want to get the top 20 documents representing each topic: documents that have the highest probability in a topic. And I want to save them in a CSV file with this format: 4 columns for Topic ID, Topic words, probability of each word in the topic, top 20 documents for each topic.

I have tried get_document_topics which I think it is the best approach for this task:

all_topics = lda_model.get_document_topics(corpus, minimum_probability=0.0, per_word_topics=False)

But I am not sure how to get top 20 documents that best represent the topic and add them to the CSV file.

    data_words_nostops = remove_stopwords(processed_docs)
    # Create Dictionary
    id2word = corpora.Dictionary(data_words_nostops)
    # Create Corpus
    texts = data_words_nostops
    # Term Document Frequency
    corpus = [id2word.doc2bow(text) for text in texts]
    # Build LDA model
    lda_model = gensim.models.ldamodel.LdaModel(corpus=corpus,
                                               id2word=id2word,
                                               num_topics=20,
                                               random_state=100,
                                               update_every=1,
                                               chunksize=100,
                                               passes=10,
                                               alpha='auto',
                                               per_word_topics=True)

    pprint(lda_model.print_topics())
    #save csv
    fn = "topic_terms5.csv"
    if (os.path.isfile(fn)):
        m = "a"
    else:
        m = "w"

    num_topics=20
    # save topic, term, prob data in the file
    with open(fn, m, encoding="utf8", newline='') as csvfile:
        fieldnames = ["topic_id", "term", "prob"]
        writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
        if (m == "w"):
            writer.writeheader()

        for topic_id in range(num_topics):
            term_probs = lda_model.show_topic(topic_id, topn=6)
            for term, prob in term_probs:
                row = {}
                row['topic_id'] = topic_id
                row['prob'] = prob
                row['term'] = term
                writer.writerow(row)

Expected result: CSV file with this format: 4 columns for Topic ID, Topic words, probability of each word, top 20 documents for each topic.

Answer 1

First, each document has a topic vector, a list of tuples looking like this:

[(0, 3.0161273e-05), (1, 3.0161273e-05), (2, 3.0161273e-05), (3, 3.0161273e-05), (4, 
3.0161273e-05), (5, 0.06556476), (6, 0.14744747), (7, 3.0161273e-05), (8, 3.0161273e- 
05), (9, 3.0161273e-05), (10, 3.0161273e-05), (11, 0.011416071), (12, 3.0161273e-05), 
(13, 3.0161273e-05), (14, 3.0161273e-05), (15, 0.057074558), (16, 3.0161273e-05), 
(17, 3.0161273e-05), (18, 3.0161273e-05), (19, 3.0161273e-05), (20, 0.7178939), (21, 
 3.0161273e-05), (22, 3.0161273e-05), (23, 3.0161273e-05), (24, 3.0161273e-05)]

Where, for example, (0, 3.0161273e-05), 0 is the topic ID, and 3.0161273e-05 is the probability.

You need to rearrange this data structure into a form so that you can compare across documents.

Here is what you can do:

#Create a dictionary, with topic ID as the key, and the value is a list of tuples 
(docID, probability of this particular topic for the doc) 

topic_dict = {i: [] for i in range(20)}  # Assuming you have 20 topics. 

#Loop over all the documents to group the probability of each topic

for docID in range(num_docs):
    topic_vector = lda_model[corpus[docID]]
    for topicID, prob in topic_vector:
        topic_dict[topicID].append((docID, prob))

#Then, you can sort the dictionary to find the top 20 documents:

for topicID, probs in topic_dict.items():
    doc_probs = sorted(probs, key = lambda x: x[1], reverse = True)
    docs_top_20 = [dp[0] for dp in doc_probs[:20]]  

You get the topic 20 docs for each topic. You can collect in a list (this will be a list of lists) or a dictionary, so that they can be outputted.