具有两个单词标记的文本分类

Question

I'm trying to do some key information extraction using NLTK and word tokenization in advertisements.

For example: "The room has max capacity of 800 people no smoking allowed no children above 12 yr old ..."

My question is: - (max capacity) has a meaning different than capacity. - no smoking is different than smoking.

How can I tokenize composite words to analyse? I don't want to break ["no","smoking"] I want to have a token ["no smoking"]

word_tokenize(text)

Also, when I tokenize and remove stop words I lose the negative meaning of the words.

Answer 1

I think what you're looking for is NLTK's ngrams

from nltk import ngrams

text = "The room has max capacity of 800 people no smoking allowed no children above 12 yr old ..."

pairs = ngrams(text.split(), 2) # change the 2 here to however many words you want in each group

for pair in pairs:
    print(pair)

> ('The', 'room')
('room', 'has')
('has', 'max')
('max', 'capacity')
('capacity', 'of')
('of', '800')
('800', 'people')
('people', 'no')
('no', 'smoking')
('smoking', 'allowed')
('allowed', 'no')
('no', 'children')
('children', 'above')
('above', '12')
('12', 'yr')
('yr', 'old')
('old', '...')

Hope this helps

Edit:

If you're then going to use TF-IDF may i recommend sklearn.feature_extraction.text.TfidfVectorizer which has ngram_range as a parameter ngram_range=(2, 2) would give you the pairs that you're after, meaning you don't need to use the above code before hand.