Gensim topic printing errors/issues

Question

All,

This is a re-post to what I responded to over in this thread . I am getting some totally screwy results with trying to print LSI topics in gensim. Here is my code:

try:
    from gensim import corpora, models
except ImportError as err:
    print err

class LSI:
    def topics(self, corpus):
        tfidf = models.TfidfModel(corpus)
        corpus_tfidf = tfidf[corpus]
        dictionary = corpora.Dictionary(corpus)
        lsi = models.LsiModel(corpus_tfidf, id2word=dictionary, num_topics=5)
        print lsi.show_topics()

if __name__ == '__main__':
    data = '../data/data.txt'
    corpus = corpora.textcorpus.TextCorpus(data)
    LSI().topics(corpus)

This prints the following to the console.

-0.804*"(5, 1)" + -0.246*"(856, 1)" + -0.227*"(145, 1)" + ......

I would like to be able to print out the topics like @2er0 did over here but I am getting results like these. See below and note that the second item that is printed is a tuple and I have no idea where it came from. data.txt is a text file with several paragraphs in it. That is all.

Any thoughts on this would be fantastic! Adam

Answer 1

To answer why your LSI topics are tuples instead of words, check your input corpus.

is it created from a list of documents that is converted into corpus through corpus = [dictionary.doc2bow(text) for text in texts] ?

Because if it isn't and you just read it from serialized corpus without reading a dictionary, then you wont get the words in the topic outputs.

Below my code works and prints out the topics with weighted words:

import gensim as gs

documents = ["Human machine interface for lab abc computer applications",
             "A survey of user opinion of computer system response time",
             "The EPS user interface management system",
             "System and human system engineering testing of EPS",
             "Relation of user perceived response time to error measurement",
             "The generation of random binary unordered trees",
             "The intersection graph of paths in trees",
             "Graph minors IV Widths of trees and well quasi ordering",
             "Graph minors A survey"]

texts = [[word for word in document.lower().split()] for document in documents]
dictionary = gs.corpora.Dictionary(texts)
corpus = [dictionary.doc2bow(text) for text in texts]

tfidf = gs.models.TfidfModel(corpus)
corpus_tfidf = tfidf[corpus]

lsi = gs.models.LsiModel(corpus_tfidf, id2word=dictionary, num_topics=5)
lsi.print_topics()

for i in lsi.print_topics():
    print i

The above outputs:

-0.331*"system" + -0.329*"a" + -0.329*"survey" + -0.241*"user" + -0.234*"minors" + -0.217*"opinion" + -0.215*"eps" + -0.212*"graph" + -0.205*"response" + -0.205*"time"
-0.330*"minors" + 0.313*"eps" + 0.301*"system" + -0.288*"graph" + -0.274*"a" + -0.274*"survey" + 0.268*"management" + 0.262*"interface" + 0.208*"human" + 0.189*"engineering"
0.282*"trees" + 0.267*"the" + 0.236*"in" + 0.236*"paths" + 0.236*"intersection" + -0.233*"time" + -0.233*"response" + 0.202*"generation" + 0.202*"unordered" + 0.202*"binary"
-0.247*"generation" + -0.247*"unordered" + -0.247*"random" + -0.247*"binary" + 0.219*"minors" + -0.214*"the" + -0.214*"to" + -0.214*"error" + -0.214*"perceived" + -0.214*"relation"
0.333*"machine" + 0.333*"for" + 0.333*"lab" + 0.333*"abc" + 0.333*"applications" + 0.258*"computer" + -0.214*"system" + -0.194*"eps" + -0.191*"and" + -0.188*"testing"

Answer 2

It looks ugly but this does the job (just a purely string based approach):

#x = lsi.show_topics()
x = '-0.804*"(5, 1)" + -0.246*"(856, 1)" + -0.227*"(145, 1)"'
y = [(j.split("*")[0], (j.split("*")[1].split(",")[0].lstrip('"('), j.split("*")[1].split(",")[1].strip().rstrip(')"'))) for j in [i for i in x.strip().split(" + ")]]

for i in y:
  print y

The above outputs:

('-0.804', ('5', '1'))
('-0.246', ('856', '1'))
('-0.227', ('145', '1'))

If not you can try lsi.print_topic(i) instead of lsi.show_topics()

for i in range(len(lsi.show_topics())):
  print lsi.print_topic(i)