如何通过gensim在LDA分析中删除单词

Question

I'm using gensim to do a LDA topic modeling work. My data was pretreated by some other people. He gave me two things. ①the mmcorpus file(imported by gensim.corpora.MmCorpus function) ②the dictionary file(imported by gensim.corpora.Dictionary.load function) I created the LDA model successfully and adjusted the superparameter ALPHA from 0.5-1.5 and I drew a visualized chart like this: I was confused why there are several tall bars there. And I found some strange words like this: Interestingly the letter "b" which I haven't seen before appears. The man who gave me the data said the letter "b" may generated automatically when he converted the data into bytes type. He doesn't know how to erase the "b" neither do I. How can I delete the "b" when I just have the mmcorpus file and the dictionary file? Please!

Answer 1

gensim has a function for filtering out specific tokens from the dictionary. You just have to know their corresponding ID. As for the corpus, I am not aware of any built-in functions that let you modify its content. You can however convert the (usually sparse) corpus to a dense numpy array, delete a column and convert it back to MmCorpus format. After that, you should be able to use both the modified dictionary and corpus to train a new LDA model, this time without the unwanted words. Here is my shot at it with a small toy corpus:

import gensim
import numpy as np

# toy document set
texts = ['This is my first b', 'Another b just like so']
tokenlist = [list(gensim.utils.tokenize(text)) for text in texts]

# create dictionary and MmCorpus
dictionary = gensim.corpora.Dictionary(tokenlist)
corpus = [dictionary.doc2bow(tokens) for tokens in tokenlist]
gensim.corpora.MmCorpus.serialize('MmCorpusTest.mm', corpus)

# assume the word 'b' is to be deleted, put its id in a variable
del_ids = [k for k,v in dictionary.items() if v=='b']

# remove unwanted word ids from the dictionary in place
dictionary.filter_tokens(bad_ids=del_ids)

# load corpus from your file
corpusMm = gensim.corpora.MmCorpus('MmCorpusTest.mm')
# convert corpus to a dense array, transpose because by default documents would be columns
np_corpus = gensim.matutils.corpus2dense(corpusMm, corpusMm.num_terms, num_docs=corpusMm.num_docs).T
# delete columns for specified tokens, transpose back afterwards
np_corpus = np.delete(np_corpus, del_ids, 1).T
# convert array to corpus
new_corpus = gensim.matutils.Dense2Corpus(np_corpus)