如何解释ntlk包中的“信息最丰富”

Question

I'm new to NLP and am struggling to interpret the results I get when looking at a simple example of NLP classification of most important features. Specifically, in the common example I've shown below, I don't understand why the word "this" is informative when it appears in 3/5 negative sentiment sentences, and 3/5 positive sentences?

train = [('I love this sandwich.', 'pos'),
('This is an amazing place!', 'pos'),
('I feel very good about these beers.', 'pos'),
('This is my best work.', 'pos'),
("What an awesome view", 'pos'),
('I do not like this restaurant', 'neg'),
('I am tired of this stuff.', 'neg'),
("I can't deal with this", 'neg'),
('He is my sworn enemy!', 'neg'),
('My boss is horrible.', 'neg')]

from nltk.tokenize import word_tokenize # or use some other tokenizer
all_words = set(word.lower() for passage in train for word in word_tokenize(passage[0]))
t = [({word: (word in word_tokenize(x[0])) for word in all_words}, x[1]) for x in train]

import nltk
classifier = nltk.NaiveBayesClassifier.train(t)
classifier.show_most_informative_features()

Here are the results:

Most Informative Features
                    this = True              neg : pos    =      2.3 : 1.0
                    this = False             pos : neg    =      1.8 : 1.0
                      an = False             neg : pos    =      1.6 : 1.0
                       . = False             neg : pos    =      1.4 : 1.0
                       . = True              pos : neg    =      1.4 : 1.0
                    feel = False             neg : pos    =      1.2 : 1.0
                      of = False             pos : neg    =      1.2 : 1.0
                     not = False             pos : neg    =      1.2 : 1.0
                      do = False             pos : neg    =      1.2 : 1.0
                    very = False             neg : pos    =      1.2 : 1.0

Any ideas? I'd love an explanation of what the formula is that calculates the probability of a word / its informativeness.

I also did this super simple example:

train = [('love', 'pos'),
('love', 'pos'),
('love', 'pos'),
('bad', 'pos'),
("bad", 'pos'),
('bad', 'neg'),
('bad', 'neg'),
("bad", 'neg'),
('bad', 'neg'),
('love', 'neg')]

And get the following:

Most Informative Features
                     bad = False             pos : neg    =      2.3 : 1.0
                    love = True              pos : neg    =      2.3 : 1.0
                    love = False             neg : pos    =      1.8 : 1.0
                     bad = True              neg : pos    =      1.8 : 1.0

Which while directionally right doesn't seem to match up with any likelihood ratio calculation I can figure out.

Answer 1

Looking at the source of show_most_informative_features() , from nltk's documentation ,

Informativeness of a feature (fname,fval) is equal to the highest value of P(fname=fval|label), for any label, divided by the lowest value of P(fname=fval|label), for any label.

But, for your case, there is simply not enough data points to calculate these probabilities, ie the probability distribution is more or less flat, which you can see from the raw weight values of the features. That is probably why irrelevant features are marked as most important. If you experiment by adding just 3-4 more sentences, will notice how this changes.