I have made doc2vec file by training data using gensim model now while processing it. I am getting an error. I am running the below code:-
model = Doc2Vec.load('sentiment140.d2v')
if len(sys.argv) < 4:
print ("Please input train_pos_count, train_neg_count and classifier!")
sys.exit()
train_pos_count = int(sys.argv[1])
train_neg_count = int(sys.argv[2])
test_pos_count = 144
test_neg_count = 144
print (train_pos_count)
print (train_neg_count)
vec_dim = 100
print ("Build training data set...")
train_arrays = numpy.zeros((train_pos_count + train_neg_count, vec_dim))
train_labels = numpy.zeros(train_pos_count + train_neg_count)
for i in range(train_pos_count):
prefix_train_pos = 'TRAIN_POS_' + str(i)
train_arrays[i] = model.docvecs[prefix_train_pos]
train_labels[i] = 1
for i in range(train_neg_count):
prefix_train_neg = 'TRAIN_NEG_' + str(i)
train_arrays[train_pos_count + i] = model.docvecs[prefix_train_neg]
train_labels[train_pos_count + i] = 0
print ("Build testing data set...")
test_arrays = numpy.zeros((test_pos_count + test_neg_count, vec_dim))
test_labels = numpy.zeros(test_pos_count + test_neg_count)
for i in range(test_pos_count):
prefix_test_pos = 'TEST_POS_' + str(i)
test_arrays[i] = model.docvecs[prefix_test_pos]
test_labels[i] = 1
for i in range(test_neg_count):
prefix_test_neg = 'TEST_NEG_' + str(i)
test_arrays[test_pos_count + i] = model.docvecs[prefix_test_neg]
test_labels[test_pos_count + i] = 0
print ("Begin classification...")
classifier = None
if sys.argv[3] == '-lr':
print ("Logistic Regressions is used...")
classifier = LogisticRegression()
elif sys.argv[3] == '-svm':
print ("Support Vector Machine is used...")
classifier = SVC()
elif sys.argv[3] == '-knn':
print ("K-Nearest Neighbors is used...")
classifier = KNeighborsClassifier(n_neighbors=10)
elif sys.argv[3] == '-rf':
print ("Random Forest is used...")
classifier = RandomForestClassifier()
classifier.fit(train_arrays, train_labels)
print ("Accuracy:", classifier.score(test_arrays, test_labels))
I am getting a Keyerror - "TEST_POS_72"
I want to know what I am doing wrong.
The error means quite literally that no doc-vector with the key ('tag') TEST_POS_72
is part of the model. There mustn't have been any documents with that tag presented during training.
You can see a list of all known doc-tags in the model in model.docvecs.offset2doctag
. If TEST_POS_72
isn't there, you can't access a doc-vector via model.docvecs['TEST_POS_72']
. (If that list is empty, then the doc-vectors were trained to be accessed by plain int keys – and model.docvecs[72]
would be a more appropriate way to access a doc-vector.)
(Separately, Doc2Vec won't work well with tiny corpuses of a few hundred documents, and the warning in your screenshot "Slow version of gensim.models.doc2vec is being used" means that gensim's optimized C-compiled routines weren't part of the installation, and training will be 100x or more slower.)
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.