[英]Create Dictionary from Penn Treebank Corpus sample from NLTK?
I know that the Treebank corpus is already tagged, but unlike the Brown corpus, I can't figure out how to get a dictionary of tags. 我知道Treebank语料库已被标记,但与布朗语料库不同,我无法弄清楚如何获取标签词典。 For instance,
例如,
>>> from nltk.corpus import brown
>>> wordcounts = nltk.ConditionalFreqDist(brown.tagged_words())
This doesn't work on the Treebank corpus? 这不适用于Treebank语料库?
Quick solution: 快速解决方案
>>> from nltk.corpus import treebank
>>> from nltk import ConditionalFreqDist as cfd
>>> from itertools import chain
>>> treebank_tagged_words = list(chain(*list(chain(*[[tree.pos() for tree in treebank.parsed_sents(pf)] for pf in treebank.fileids()]))))
>>> wordcounts = cfd(treebank_tagged_words)
>>> treebank_tagged_words[0]
(u'Pierre', u'NNP')
>>> wordcounts[u'Pierre']
FreqDist({u'NNP': 1})
>>> treebank_tagged_words[100]
(u'asbestos', u'NN')
>>> wordcounts[u'asbestos']
FreqDist({u'NN': 11})
For more details, see https://en.wikipedia.org/wiki/User:Alvations/NLTK_cheatsheet/CorporaReaders#Penn_Tree_Bank 有关详细信息,请参阅https://en.wikipedia.org/wiki/User:Alvations/NLTK_cheatsheet/CorporaReaders#Penn_Tree_Bank
See also: Is there a way of avoiding so many list(chain(*list_of_list))? 另请参阅: 有没有办法避免这么多列表(chain(* list_of_list))?
Note that there are only 3000+ sentences from the Penn Treebank sample from NLTK, the brown corpus has 50,000 sentences. 请注意,来自NLTK的Penn Treebank样本中只有3000多个句子,棕色语料库有50,000个句子。
To split the sentences up into training and test set: 将句子分成训练和测试集:
from nltk.corpus import treebank
from nltk import ConditionalFreqDist as cfd
from itertools import chain
treebank_tagged_sents = list(chain(*[[tree.pos() for tree in treebank.parsed_sents(pf)] for pf in treebank.fileids()]))
total_len = len(treebank_tagged_sents)
train_len = int(90 * total_len /100)
train_set = treebank_tagged_sents[:train_len]
print len(train_set)
train_treebank_tagged_words = cfd(chain(*train_set))
test_set = treebank_tagged_sents[train_len:]
print len(test_set)
test_treebank_tagged_words = cfd(chain(*test_set))
If you're going to use brown corpus (that does not contain parsed sentence), you can used the tagged_sent()
: 如果您要使用棕色语料库(不包含已解析的句子),您可以使用
tagged_sent()
:
>>> from nltk.corpus import brown
>>> brown_tagged_sents = brown.tagged_sents()
>>> len(brown_tagged_sents)
57340
>>> brown_tagged_sents[0]
[(u'The', u'AT'), (u'Fulton', u'NP-TL'), (u'County', u'NN-TL'), (u'Grand', u'JJ-TL'), (u'Jury', u'NN-TL'), (u'said', u'VBD'), (u'Friday', u'NR'), (u'an', u'AT'), (u'investigation', u'NN'), (u'of', u'IN'), (u"Atlanta's", u'NP$'), (u'recent', u'JJ'), (u'primary', u'NN'), (u'election', u'NN'), (u'produced', u'VBD'), (u'``', u'``'), (u'no', u'AT'), (u'evidence', u'NN'), (u"''", u"''"), (u'that', u'CS'), (u'any', u'DTI'), (u'irregularities', u'NNS'), (u'took', u'VBD'), (u'place', u'NN'), (u'.', u'.')]
>>> total_len = len(brown_tagged_sents)
>>> train_len = int(90 * total_len/100)
>>> train_set = brown_tagged_sents[:train_len]
>>> train_brown_tagged_words = cfd(chain(*train_set))
>>> train_brown_tagged_words['asbestos']
FreqDist({u'NN': 1})
As @alexis noted, unless you're splitting the corpus at sentence level. 正如@alexis指出的那样,除非你在语句层面拆分语料库。 The
tagged_words()
function also exist in the Penn Treebank API in NLTK: tagged_words()
函数也存在于NLTK中的Penn Treebank API中:
>>> from nltk.corpus import treebank
>>> from nltk.corpus import brown
>>> treebank.tagged_words()
[(u'Pierre', u'NNP'), (u'Vinken', u'NNP'), ...]
>>> brown.tagged_words()
[(u'The', u'AT'), (u'Fulton', u'NP-TL'), ...]
>>> type(treebank.tagged_words())
<class 'nltk.corpus.reader.util.ConcatenatedCorpusView'>
>>> type(brown.tagged_words())
<class 'nltk.corpus.reader.util.ConcatenatedCorpusView'>
>>> from nltk import ConditionalFreqDist as cfd
>>> cfd(brown.tagged_words())
<ConditionalFreqDist with 56057 conditions>
>>> cfd(treebank.tagged_words())
<ConditionalFreqDist with 12408 conditions>
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.