在Python中创建多层字典POS标记器

Question

I have a text file with POS tags. For example:

"DT The NN dog VB jumps..."

I need to create a dictionary where the keys of the entries are the words and the values are another dictionary with the tags as keys and the frequency of the tags as values. So what I would need would look like this:

{'The': {'DT': 47}}, {'dog': {'VB': 32}} ...

I'm at a total loss right now. I've started by taking my text file and splitting it into a list of strings with, so that it is a list like

'DT The' 'NN dog' 'VB jumps'

I'm not sure if this is even the right first step or what. Please help!

Answer 1

This approach should give you the structure you're looking for, with the POS counts being the full count of that tag within the corpus presented.

NOTE: The RETAIN_PUNCTUATION_FLAG and RETAIN_CASE_FLAG allow you to toggle the behavior to either strip punctuation before evaluation, make the case uniform or retain upper/lower casing, or simply do both. Here, they're both assigned False , all words will be handled as lowercase, and all ASCII punctuation will be stripped before evaluation.

I've added word_list and pos_list for alternative listing.

from string import punctuation

RETAIN_PUNCTUATION_FLAG = False
RETAIN_CASE_FLAG = False

string = "DT The NN dog VB jumps DT the NN sofa. DT The NN cat VB pages DT the NN page."

punctuation_strip_table = str.maketrans('', '', punctuation)
if RETAIN_CASE_FLAG and RETAIN_PUNCTUATION_FLAG:
    pass
elif RETAIN_CASE_FLAG and not RETAIN_PUNCTUATION_FLAG:
    string = string.translate(punctuation_strip_table)
elif not RETAIN_CASE_FLAG and RETAIN_PUNCTUATION_FLAG:
    string = string.casefold()
elif not RETAIN_CASE_FLAG and not RETAIN_PUNCTUATION_FLAG:
    string = string.casefold().translate(punctuation_strip_table)

list_all = string.split(' ')
pos_word_pairs = set(zip(
            list_all[0:][::2],
            list_all[1:][::2]))

pos_list = {pos.upper(): {
    'count': list_all.count(pos),
    'words': [
        word
        for match_pos, word in pos_word_pairs
        if match_pos == pos]
    }
    for pos in set(list_all[0:][::2])}
word_list = {word: {
    'count': list_all.count(word),
    'pos': [
        pos.upper()
        for pos, match_word in pos_word_pairs
        if match_word == word]
    }
    for word in set(list_all[1:][::2])}
paired = {
        word: {
            pos.upper():
            list_all.count(pos)}
        for pos, word
        in pos_word_pairs}

print('pos_list:', pos_list)
print()
print('word_list:', word_list)
print()
print('paired:',paired)

Output:

pos_list: {'VB': {'count': 2, 'words': ['pages', 'jumps']}, 'NN': {'count': 4, 'words': ['page', 'dog', 'sofa', 'cat']}, 'DT': {'count': 4, 'words': ['the']}}

word_list: {'dog': {'count': 1, 'pos': ['NN']}, 'cat': {'count': 1, 'pos': ['NN']}, 'jumps': {'count': 1, 'pos': ['VB']}, 'the': {'count': 4, 'pos': ['DT']}, 'page': {'count': 1, 'pos': ['NN']}, 'sofa': {'count': 1, 'pos': ['NN']}, 'pages': {'count': 1, 'pos': ['VB']}}

paired: {'pages': {'VB': 2}, 'jumps': {'VB': 2}, 'the': {'DT': 4}, 'page': {'NN': 4}, 'dog': {'NN': 4}, 'sofa': {'NN': 4}, 'cat': {'NN': 4}}