情绪分类与监督学习

Question

I am doing sentiment classification on blogs from Livejournal with python's scikit-learn. I have around 40000 posts and I use 4/5 of them as training set and the remains as test set.

There are 6 sentiments: ['joy','sadness','anger','surprise','love','fear']

I experiented with several classifiers(including naive bayes,svm,sgd..) but the problem is that the prediction is very very inaccurate. Actually it is nearly trivial, because almost every blog in the test set is predicted to 'joy', which is the most frequent sentiment in the train set (45%).

The feature set basicly includes bag of words features(I tried unigrams and bigrams), for unigram there are 613822 features in total.

Besiedes, I added some lexicon-based features using SentiWordnet scores: calculating sum of positive and negative scores of nouns,adjs,ajvs,verbs and total words. in a blog. So for each blog, there will be 613822 + 5 features.

I also applied some feature selection methods such as chi2 to reduce the feature number, but there isn't any apparent improvments.

scikit-learn's CountVectorizer and DictVectorizer are used for vectorize the features, and Pipeline.FeatureUnion is used for concatenating them.

I guess the poor result is due to the overly large bag of words feature set -- maybe there are too many misspelled words in the text? (I already removed the stops words and done some lemmatization jobs) I also think the lexicon-based features don't really work because the BOW feature is too large.

I'm hoping to find any obvious error in my approach or what I can do to improve the accuracy.

Thanks for any advice!!

Answer 1

You are right, The problem is in this overly large number of features and you're over-fitting to it..

Consider the following:

1- Normalize the each blog, remove numbers, punctuation, links, html tags if any.

2- Consider Stemming instead of Lemmatization, Stemmers are much simpler, smaller and usually faster than lemmatizers, and for many applications their results are good enough.

http://nlp.stanford.edu/IR-book/html/htmledition/stemming-and-lemmatization-1.html

Stemming usually refers to a crude heuristic process that chops off the ends of words in the hope of achieving this goal correctly most of the time, and often includes the removal of derivational affixes. Lemmatization usually refers to doing things properly with the use of a vocabulary and morphological analysis of words, normally aiming to remove inflectional endings only

3- I worked in a similar problem before, What I did for features extraction, is for every sentiment of the 6 sentiments, I got the most frequent 500 words (for each class), then removed the common stems between their union. The resulting list contained about 2000 words which then used as features list. Then I used Naive Bayes classifier.