[英]KNN for Text Classification using TF-IDF scores
我在语料库中有一个 CSV 文件 (corpus.csv),其中包含以下格式的分级摘要(文本):
Institute, Score, Abstract
----------------------------------------------------------------------
UoM, 3.0, Hello, this is abstract one
UoM, 3.2, Hello, this is abstract two and yet counting.
UoE, 3.1, Hello, yet another abstract but this is a unique one.
UoE, 2.2, Hello, please no more abstract.
我正在尝试在 python 中创建一个 KNN 分类程序,该程序能够获取用户输入摘要,例如“这是一个新的唯一摘要”,然后对最接近语料库 (CSV) 的用户输入摘要进行分类,并返回预测摘要的分数/等级。 我怎样才能做到这一点?
我有以下代码:
from sklearn.feature_extraction.text import TfidfVectorizer
from nltk.corpus import stopwords
import numpy as np
import pandas as pd
from csv import reader,writer
import operator as op
import string
#Read data from corpus
r = reader(open('corpus.csv','r'))
abstract_list = []
score_list = []
institute_list = []
row_count = 0
for row in list(r)[1:]:
institute,score,abstract = row
if len(abstract.split()) > 0:
institute_list.append(institute)
score = float(score)
score_list.append(score)
abstract = abstract.translate(string.punctuation).lower()
abstract_list.append(abstract)
row_count = row_count + 1
print("Total processed data: ", row_count)
#Vectorize (TF-IDF, ngrams 1-4, no stop words) using sklearn -->
vectorizer = TfidfVectorizer(analyzer='word', ngram_range=(1,4),
min_df = 0, stop_words = 'english', sublinear_tf=True)
response = vectorizer.fit_transform(abstract_list)
feature_names = vectorizer.get_feature_names()
在上述代码中,如何将 TF-IDF 计算中的特征用于上述 KNN 分类? (可能使用 sklearn.neighborsKNeighborsClassifier 框架)
PS 此应用案例的类别是摘要的相应分数/等级。
我有视觉深度学习的背景,但是,我缺乏文本分类方面的知识,尤其是使用 KNN。 任何帮助将非常感激。 先感谢您。
KNN 是一种分类算法——这意味着你必须有一个类属性。 KNN 可以使用 TFIDF 的输出作为输入矩阵 - TrainX,但您仍然需要 TrainY - 数据中每一行的类。 但是,您可以使用 KNN 回归器。 使用您的分数作为班级变量:
from sklearn.feature_extraction.text import TfidfVectorizer
from nltk.corpus import stopwords
import numpy as np
import pandas as pd
from csv import reader,writer
import operator as op
import string
from sklearn import neighbors
#Read data from corpus
r = reader(open('corpus.csv','r'))
abstract_list = []
score_list = []
institute_list = []
row_count = 0
for row in list(r)[1:]:
institute,score,abstract = row[0], row[1], row[2]
if len(abstract.split()) > 0:
institute_list.append(institute)
score = float(score)
score_list.append(score)
abstract = abstract.translate(string.punctuation).lower()
abstract_list.append(abstract)
row_count = row_count + 1
print("Total processed data: ", row_count)
#Vectorize (TF-IDF, ngrams 1-4, no stop words) using sklearn -->
vectorizer = TfidfVectorizer(analyzer='word', ngram_range=(1,4),
min_df = 0, stop_words = 'english', sublinear_tf=True)
response = vectorizer.fit_transform(abstract_list)
classes = score_list
feature_names = vectorizer.get_feature_names()
clf = neighbors.KNeighborsRegressor(n_neighbors=1)
clf.fit(response, classes)
clf.predict(response)
“预测”将预测每个实例的分数。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.