scikit学习矢量化器的自定义标记器

Question

给出以下文件清单：

docs = [
'feature one`feature two`feature three',
'feature one`feature two`feature four',
'feature one'
]

我想使用scikit中的两个矢量化器类（ CountVectorizer或TfidfVectorizer ），其中'feature one' ， 'feature two' ， 'feature three'和'feature four'应该是矩阵中表示的四个功能。

我尝试了这个：

vec = CountVectorizer(token_pattern='(?u)\w+\s.\w.`')

但这仅返回以下内容：

['feature one`', 'feature two`']

Answer 1

如果您已将功能固定为

'feature one', 'feature two', 'feature three', and 'feature four'

然后您还可以使用vocabulary参数。

vocab = ['feature one', 'feature two', 'feature three', 'feature four']
vec = CountVectorizer(vocabulary=vocab)

X = vec.fit_transform(docs)
vec.get_feature_names()
Out[310]:
['feature one',
 'feature two',
 'feature three',
 'feature four']

Answer 2

In [295]: vec = CountVectorizer(token_pattern='(?u)\w+[\s\`]\w+')

In [296]: X = vec.fit_transform(docs)

In [297]: vec.get_feature_names()
Out[297]: ['feature four', 'feature one', 'feature three', 'feature two']

您可能还需要考虑使用ngram_range=(2,2) ，它将产生以下结果：

In [308]: vec = CountVectorizer(ngram_range=(2,2))

In [309]: X = vec.fit_transform(docs)

In [310]: vec.get_feature_names()
Out[310]:
['feature four',
 'feature one',
 'feature three',
 'feature two',
 'one feature',
 'two feature']