解析penn語法樹以提取其語法規則

Question

我有一個PENN-Syntax-Tree，我想以遞歸方式獲取該樹包含的所有規則。

(ROOT 
(S 
   (NP (NN Carnac) (DT the) (NN Magnificent)) 
   (VP (VBD gave) (NP ((DT a) (NN talk))))
)
)

我的目標是獲得如下的語法規則：

ROOT --> S
S --> NP VP
NP --> NN
...

正如我所說，我需要遞歸地執行此操作，而無需NLTK包或任何其他模塊或正則表達式 。 這是我到目前為止所擁有的。 參數tree是在每個空間上分割的Penn-Tree。

def extract_rules(tree):
    tree = tree[1:-1]
    print("\n\n")

    if len(tree) == 0:
        return

    root_node = tree[0]
    print("Current Root: "+root_node)

    remaining_tree = tree[1:]
    right_side = []

    temp_tree = list(remaining_tree)
    print("remaining_tree: ", remaining_tree)
    symbol = remaining_tree.pop(0)

    print("Symbol: "+symbol)

    if symbol not in ["(", ")"]:
        print("CASE: No Brackets")
        print("Rule: "+root_node+" --> "+str(symbol))

        right_side.append(symbol)

    elif symbol == "(":
        print("CASE: Opening Bracket")
        print("Temp Tree: ", temp_tree)
        cursubtree_end = bracket_depth(temp_tree)
        print("Subtree ends at position "+str(cursubtree_end)+" and Element is "+temp_tree[cursubtree_end])
        cursubtree_start = temp_tree.index(symbol)

        cursubtree = temp_tree[cursubtree_start:cursubtree_end+1]
        print("Subtree: ", cursubtree)

        rnode = extract_rules(cursubtree)
        if rnode:
            right_side.append(rnode)
            print("Rule: "+root_node+" --> "+str(rnode))

    print(right_side)
    return root_node


def bracket_depth(tree):
    counter = 0
    position = 0
    subtree = []

    for i, char in enumerate(tree):
        if char == "(":
            counter = counter + 1
        if char == ")":
            counter = counter - 1

        if counter == 0 and i != 0:
            counter = i
            position = i
            break

    subtree = tree[0:position+1]

    return position

目前它適用於S的第一個子樹，但所有其他子樹都不會被遞歸解析。 很高興得到任何幫助..

Answer 1

我傾向於保持盡可能簡單，而不是試圖重新發明你目前不允許使用的解析模塊。 就像是：

string = '''
    (ROOT
        (S
            (NP (NN Carnac) (DT the) (NN Magnificent))
            (VP (VBD gave) (NP (DT a) (NN talk)))
        )
    )
'''

def is_symbol_char(character):
    '''
    Predicate to test if a character is valid
    for use in a symbol, extend as needed.
    '''

    return character.isalpha() or character in '-=$!?.'

def tokenize(characters):
    '''
    Process characters into a nested structure.  The original string
    '(DT the)' is passed in as ['(', 'D', 'T', ' ', 't', 'h', 'e', ')']
    '''

    tokens = []

    while characters:
        character = characters.pop(0)

        if character.isspace():
            pass  # nothing to do, ignore it

        elif character == '(':  # signals start of recursive analysis (push)
            characters, result = tokenize(characters)
            tokens.append(result)

        elif character == ')':  # signals end of recursive analysis (pop)
            break

        elif is_symbol_char(character):
            # if it looks like a symbol, collect all
            # subsequents symbol characters
            symbol = ''

            while is_symbol_char(character):
                symbol += character
                character = characters.pop(0)

            # push unused non-symbol character back onto characters
            characters.insert(0, character)

            tokens.append(symbol)

    # Return whatever tokens we collected and any characters left over
    return characters, tokens

def extract_rules(tokens):
    ''' Recursively walk tokenized data extracting rules. '''

    head, *tail = tokens

    print(head, '-->', *[x[0] if isinstance(x, list) else x for x in tail])

    for token in tail:  # recurse
        if isinstance(token, list):
            extract_rules(token)

characters, tokens = tokenize(list(string))

# After a successful tokenization, all the characters should be consumed
assert not characters, "Didn't consume all the input!"

print('Tokens:', tokens[0], 'Rules:', sep='\n\n', end='\n\n')

extract_rules(tokens[0])

OUTPUT

Tokens:

['ROOT', ['S', ['NP', ['NN', 'Carnac'], ['DT', 'the'], ['NN', 'Magnificent']], ['VP', ['VBD', 'gave'], ['NP', ['DT', 'a'], ['NN', 'talk']]]]]

Rules:

ROOT --> S
S --> NP VP
NP --> NN DT NN
NN --> Carnac
DT --> the
NN --> Magnificent
VP --> VBD NP
VBD --> gave
NP --> DT NN
DT --> a
NN --> talk

注意

我更改了原始樹作為此子句：

(NP ((DT a) (NN talk)))

似乎不正確，因為它在網絡上可用的語法樹grapher上生成一個空節點，所以我簡化為：

(NP (DT a) (NN talk))

根據需要調整。

Answer 2

這可以以更簡單的方式完成。 鑒於我們知道我們的語法結構是CNF LR，我們可以使用遞歸正則表達式解析器來解析文本。

有一個名為pyparser的軟件包（你可以使用pip install pyparser安裝它，如果你還沒有它）。

from pyparsing import nestedExpr

astring = '''(ROOT 
(S 
   (NP (NN Carnac) (DT the) (NN Magnificent)) 
   (VP (VBD gave) (NP ((DT a) (NN talk))))
)
)'''

expr = nestedExpr('(', ')')
result = expr.parseString(astring).asList()[0]
print(result)

這給了

['ROOT', ['S', ['NP', ['NN', 'Carnac'], ['DT', 'the'], ['NN', 'Magnificent']], ['VP', ['VBD', 'gave'], ['NP', [['DT', 'a'], ['NN', 'talk']]]]]]

因此，我們已成功將字符串轉換為列表層次結構。 現在我們需要編寫一些代碼來解析列表並提取規則。

def get_rules(result, rules):
    for l in result[1:]:
        if isinstance(l, list) and not isinstance(l[0], list):
            rules.add((result[0], l[0]))  
            get_rules(l, rules)

        elif isinstance(l[0], list):
            rules.add((result[0], tuple([x[0] for x in l])))
        else:
            rules.add((result[0], l))

    return rules

正如我所提到的，我們已經知道了語法的結構，因此我們只需要處理有限的條件。

像這樣調用這個函數：

rules = get_rules(result, set()) # results was obtained from before

for i in rules:
   print i

輸出：

('ROOT', 'S')
('VP', 'NP')
('DT', 'the')
('NP', 'NN')
('NP', ('DT', 'NN'))
('NP', 'DT')
('S', 'VP')
('VBD', 'gave')
('NN', 'Carnac')
('NN', 'Magnificent')
('S', 'NP')
('VP', 'VBD')

根據需要訂購。