使用FeatureUnion在scikit学习管道中的单词袋中添加额外功能

Question

我已经尽了很多努力，但是仍然无法弄清楚如何在scikit-learn管道中的FeatureUnion同时使用文本功能和FeatureUnion功能。
我有一个句子列表及其标签以训练模型，还有一个句子列表作为测试数据。 然后，我尝试为袋式单词添加一个额外的功能（如每个句子的长度）。 为此，我编写了一个自定义的LengthTransformer ，它返回一个长度列表，并且元素数量与我的火车列表相同。
然后，我将其与使用FeatureUnion的TfidfVectorizer结合使用，但它不起作用。

到目前为止，我想到的是：

from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.pipeline import Pipeline, FeatureUnion
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.svm import LinearSVC
from sklearn.multiclass import OneVsRestClassifier
from sklearn import preprocessing

class LengthTransformer(BaseEstimator, TransformerMixin):

    def fit(self, X, y=None):
        return self

    def transform(self, X):
        return [len(x) for x in X]


X_train = ["new york is a hell of a town",
            "new york was originally dutch",
            "the big apple is great",
            "new york is also called the big apple",
            "nyc is nice",
            "people abbreviate new york city as nyc",
            "the capital of great britain is london",
            "london is in the uk",
            "london is in england",
            "london is in great britain",
            "it rains a lot in london",
            "london hosts the british museum",
            "new york is great and so is london",
            "i like london better than new york"]
y_train_text = [["new york"], ["new york"], ["new york"], ["new york"], ["new york"],
                ["new york"], ["london"], ["london"], ["london"], ["london"],
                ["london"], ["london"], ["london", "new york"], ["new york", "london"]]

X_test = ['nice day in nyc',
            'welcome to london',
            'london is rainy',
            'it is raining in britian',
            'it is raining in britian and the big apple',
            'it is raining in britian and nyc',
            'hello welcome to new york. enjoy it here and london too']

lb = preprocessing.MultiLabelBinarizer()
Y = lb.fit_transform(y_train_text)

classifier = Pipeline([
    ('feats', FeatureUnion([
       ('tfidf', TfidfVectorizer()),
       ('len', LengthTransformer())
    ])),
    ('clf', OneVsRestClassifier(LinearSVC()))
])

classifier.fit(X_train, Y)
predicted = classifier.predict(X_test)
all_labels = lb.inverse_transform(predicted)

for item, labels in zip(X_test, all_labels):
    print('{} => {}'.format(item, ', '.join(labels)))

Answer 1

LengthTransformer.transform返回形状是错误的-它为每个输入文档返回一个标量，而转换器应为每个文档返回一个特征向量。 您可以通过在LengthTransformer.transform中将[[len(x)] for x in X] [len(x) for x in X]更改[len(x) for x in X] [[len(x)] for x in X]使其工作。