[英]How to create Feature Vector with top Words (Feature selection in scikit-learn)
我正在使用scikit-learn创建文档的特征向量。 我的目标是使用这些功能向量创建一个二进制分类器(Genderclassifier)。
我想将k顶单词作为功能,因此两个标签文档中k个计数最高的单词(k = 10-> 20个功能,因为有2个标签)
我的两个文档(label1document,label2document)都充满了这样的实例:
user:somename, post:"A written text which i use"
到目前为止,我的理解是,我使用两个文档中所有实例的所有文本来创建带有计数(两个标签都计数,以便我可以比较labeldata)的词汇表:
#These are my documents with all text
label1document = "car eat essen sleep sleep"
label2document = "eat sleep woman woman woman woman"
vectorizer = CountVectorizer(min_df=1)
corpus = [label1document,label2document]
#Here I create a Matrix with all the countings of the words from both documents
X = vectorizer.fit_transform(corpus)
问题1:我必须放入fit_transform中才能从两个标签中获得最多计数的单词?
X_new = SelectKBest(chi2, k=2).fit_transform( ?? )
从最后开始,我想要这样的训练数据(实例):
<label> <feature1 : value> ... <featureN: value>
问题2:如何从那里获取此培训数据?
奥利弗
import pandas as pd
# get the names of the features
features = vectorizer.get_feature_names()
# change the matrix from sparse to dense
df = pd.DataFrame(X.toarray(), columns = features)
df
它将返回:
car eat essen sleep woman
0 1 1 1 2 0
1 0 1 0 1 4
然后获得最常用的术语:
highest_frequency = df.max()
highest_frequency.sort(ascending=False)
highest_frequency
哪个会返回:
woman 4
sleep 2
essen 1
eat 1
car 1
dtype: int64
将数据保存在DataFrame
,可以很容易地将其DataFrame
为所需的格式,例如:
df.to_dict()
>>> {u'car': {0: 1, 1: 0},
u'eat': {0: 1, 1: 1},
u'essen': {0: 1, 1: 0},
u'sleep': {0: 2, 1: 1},
u'woman': {0: 0, 1: 4}}
df.to_json()
>>>'{"car":{"0":1,"1":0},"eat":{"0":1,"1":1},"essen":{"0":1,"1":0},"sleep":{"0":2,"1":1},"woman":{"0":0,"1":4}}'
df.to_csv()
>>>',car,eat,essen,sleep,woman\n0,1,1,1,2,0\n1,0,1,0,1,4\n'
这是一些有用的文档 。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.