如何在python中的gensim中获取单词的最近文档

Question

I am using the doc2vec model as follows to construct my document vectors.

from gensim.models import doc2vec
from collections import namedtuple

dataset = json.load(open(input_file))

docs = []
analyzedDocument = namedtuple('AnalyzedDocument', 'words tags')

for description in dataset:
    tags = [description[0]]
    words = description[1]
    docs.append(analyzedDocument(words, tags))

model = doc2vec.Doc2Vec(docs, vector_size = 100, window = 10, min_count = 1, workers = 4, epochs = 20)

I have seen that gensim doc2vec also includes word vectors . Suppose I have a word vector created for the word deep learning . My question is; is it possible to get the documents nearest to deep learning word vector in gensim in python?

I am happy to provide more details if needed.

Answer 1

Some Doc2Vec modes will co-train doc-vectors and word-vectors in the "same space". Then, if you have a word-vector for 'deep_learning' , you can ask for documents near that vector, and the results may be useful for you. For example:

similar_docs = d2v_model.docvecs.most_similar(
                   positive=[d2v_model.wv['deep_learning']]
               )

But:

• that's only going to be as good as your model learned 'deep_learning' as a word to mean what you think of it as

• a training set of known-good documents fitting the category 'deep_learning' (and other categories) could be better - whether you hand-curate those, or try to bootstrap from other sources (like say the Wikipedia category ' Deep Learning ' or other curated/search-result sets that you trust).

• reducing a category to a single summary point (one vector) may not be as good as having a variety of examples – many points - that all fit the category. (Relevant docs may not be a neat sphere around a summary point, but rather populate exotically-shaped regions of the doc-vector high-dimensional space.) If you have a lot of good examples of each category, you could train a classifier to then label, or rank-in-relation-to-trained-categories, any further uncategorized docs.