潜在语义索引主题的数量

Question

I'm using gensim's package to implement LSI on a corpus. My goal is to find out the most frequently occurring distinct topics that appear in the corpus.

If I don't know the number of topics that are in the corpus (I'd estimate anywhere from 5 to 20), what is the best approach in setting the number of topics that LSI should search for? Is it better to look for a large number of topics (20-30), or a small number of topics (~5)?

Answer 1

From Radim himself:

that's a good question, but unfortunately without a good answer.

It is not true that increasing the number of dimensions always improves retrieval accuracy. In fact, if you use all the dimensions (=full rank of the training matrix), LSI will give you exactly the same documents that you entered in, so LSI would become pointless.

If you're interested in the math side of it, have a look at this issue: https://github.com/piskvorky/gensim/issues/28 Otherwise, just set the dimensions to a few hundred~thousand which is the accepted standard. Or try several different choices, measure the accuracy and select dimensionality that works the best on your problem.

Best, Radim

Answer 2

This is what I do sometimes when I'm confused. Since you've already narrowed down to your topics from 5-20, you can iterate b/w some of these values and see which value fits the best.

##Declare values for N_TOPICS
for i in lda.show_topics(topics=-N_TOPICS, topn=20, log=False, formatted=True): 
  print "TOPIC {0}: {1}\n".format(count, i)