标签为NLTK中的单个单词

Question

Is there a tagger that would return a single tag for a word in whatever context it maybe?

My requirement is that I need to extract words from unstructured text where the sentences would not have a structured grammar.

POS taggers are meant to work with sentences and would return a tag for a word depending on the context of the word in that sentence. So, I would either have to use another tagger that would give me the same tag for a particular word each time or use all the possible tags for a word while chunking.

Any other solutions would be greatly appreciated. Also, how can you view all the tags that can be assigned for a particular word?

Answer 1

See: http://www.nltk.org/_modules/nltk/tag.html

In particular:

>>> from nltk.corpus import brown
>>> from nltk.tag import UnigramTagger
>>> tagger = UnigramTagger(brown.tagged_sents(categories='news')[:500])
>>> sent = ['Mitchell', 'decried', 'the', 'high', 'rate', 'of', 'unemployment']
>>> for word, tag in tagger.tag(sent):
...     print(word, '->', tag)
Mitchell -> NP
decried -> None
the -> AT
high -> JJ
rate -> NN
of -> IN
unemployment -> None

The idea of the UnigramTagger is that it always assigns the tag that was most prominent for that particular word in the training corpus. Or (just above the piece of code in the docs:

This package defines several taggers, which take a token list (typically a sentence), assign a tag to each token, and return the resulting list of tagged tokens. Most of the taggers are built automatically based on a training corpus. For example, the unigram tagger tags each word w by checking what the most frequent tag for w was in a training corpus:

Not sure if there is a built-in way to view all tags that can be assigned to a particular word. Moreover; this may theoretically be as long as the total number of tags identified, as it depends on context. If you want to get an idea; what I would do is just tag your whole vocabulary and print out your vocabulary with all different tags assigned in that particular corpus.