如何在 sklearn 中进行多词标记化？

Question

I'm looking at the tokenizers in sklearn, namely CountVectorizer and DictVectorizer . I'd like to be able to debug my token counts before performing TF-IDF. However, I'm encountering difficulty in converting my nltk.multiword tokenizer into scikit learn.

Currently, I have the following:

from nltk.tokenize import MWETokenizer

tokenizer = MWETokenizer()
tokens = ["New York", "Albany", "Buffalo", "Hudson River"]
for t in tokens:
  if t.split(" "):
    print(t.split(" "))
    tokenizer.add_mwe((t.split(" ")))
  else:
    tokenizer.add_mwe(t)


# Small corpus
corpus = [
  'This is a new document about New York and the Hudson River.',
  'This is a document about California instead.'
  ]
[tokenizer.tokenize(c.split()) for c in corpus]

And I get:

[['This', 'is', 'a', 'new', 'document', 'about', 'New_York', 'and', 'the', 'Hudson', 'River.'],
 ['This', 'is', 'a', 'document', 'about', 'California', 'instead.']]

Which needs punctuation handling but recognizes "New York" as a single token, great.

Trying to apply similar to CountVectorizer , I find...

from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer(vocabulary=tokens, lowercase=False)
# >>> CountVectorizer(vocabulary=['New York', 'Albany', 'Buffalo', 'Hudson River'])

vectorizer.fit_transform(corpus).toarray()
# array([[0, 0, 0, 0],
#        [0, 0, 0, 0]])

which is wrong. How can I get counts of my (multiword) dictionary using the CountVectorizer (and ultimately TfIDFVectorizer in sklearn?

Answer 1

You might need to specify the ngrams manually. No idea if this is the right way or not:

from sklearn.feature_extraction.text import CountVectorizer
ng_min = max(min(map(lambda x: len(x.split()), tokens)),1)
ng_max = max(map(lambda x: len(x.split()), tokens))
vectorizer = CountVectorizer(vocabulary=tokens, lowercase=False, ngram_range=(ng_min, ng_max))
vectorizer.fit_transform(corpus).toarray()

yields:

array([[1, 0, 0, 1],
       [0, 0, 0, 0]])