如何在我的案例中构建我的训练数据，以便在scikit-learn中训练分类器中的SVM？

Question

I have sentences, coming from research studies, and there manually extracted word phrases, which are the key words of the sentences I want to have. Now to build the train data for a SVM classifier I would like to vectorize the sentences together with each keywords. See code

I was thinking about a dictionary and the applying DictVectorizer from the sklearn-Library.

Code:

sklearn.feature_extraction import DictVectorizer

v = DictVectorizer()

D = [{"sentence":"the laboratory information system was evaluated", 
       "keyword":"laboratory information system"},
     {"sentence":"the electronic health record system was evaluated", 
      "keyword":"electronic health record system"}]

X = v.fit_transform(D)

print(X)

content = X.toarray()

print(content)

print(v.get_feature_names())

Results:

 (0, 1) 1.0
  (0, 3)    1.0
  (1, 0)    1.0
  (1, 2)    1.0

[[0. 1. 0. 1.]
 [1. 0. 1. 0.]]

['keyword=electronic health record system', 'keyword=laboratory information system', 'sentence=the electronic health record system was evaluated', 'sentence=the laboratory information system was evaluated']

Is this methodological correct or how can I bring together each sentence with the according manually extracted keyword for vectorizing to reveice the training data. Thanks a lot.

Answer 1

I think it's not ideal to do it this way as you are using the whole sentence as a feature. It'll become problematic for a large dataset.

For example,

D = [{"sentence":"This is sentence one", 
       "keyword":"key 1"},
     {"sentence":"This is sentence one", 
       "keyword":"key 2"},
     {"sentence":"This is sentence one", 
       "keyword":"key 3"},
     {"sentence":"This is sentence one", 
       "keyword":"key 2"},
     {"sentence":"This is sentence one", 
       "keyword":"key 1"}]

X will be

[[1. 0. 0. 0. 0. 1. 0. 0.]
 [0. 1. 0. 0. 0. 0. 0. 1.]
 [0. 0. 1. 0. 0. 0. 1. 0.]
 [0. 1. 0. 0. 1. 0. 0. 0.]
 [1. 0. 0. 1. 0. 0. 0. 0.]]

You could probably just apply TfidfVectorizer from scikit-learn which'll probably pick up the important words in a sentence.

Code:

from sklearn.feature_extraction.text import TfidfVectorizer


sentences = [d['sentence'] for d in D]

vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(sentences)