如何使用 conllu 库创建 TokenList？

Question

I'm trying to create a CoNLL-U file using the conllu library as part of a Universal Dependency tagging project I'm working on.

I have a number of sentences in python lists. These contain sub-lists of tokens, lemmata, POS tags, features, etc. For example:

sentence = [['The', 'the', 'DET', ... ], ['big', big', 'ADJ', ... ], ['dog', 'dog', 'NOUN', ...], ...]

I want to automate the process of turning these into CoNLL-U parsed sentences, so I wrote the following function:

from collections import OrderedDict

def compile_sent(sent):
    sent_list = list()
    for i, tok_data in enumerate(sent):
        tok_id = i + 1
        tok = tok_data[0]
        lemma = tok_data[1]
        pos = tok_data[2]
        feats = tok_data[3]
        compiled_tok = OrderedDict({'id': tok_id, 'form': tok, 'lemma': lemma, 'upostag': pos, 'xpostag': None, 'feats': feats, 'head': None, 'deprel': None, 'deps': None, 'misc': None})
        sent_list.append(compiled_tok)
    sent_list = sent_list.serialize()
    return sent_list

print(compile_sent(sentence))

When I try to run this code I get the following error:

Traceback (most recent call last):
  File "/Users/me/PycharmProjects/UDParser/Rough_Work.py", line 103, in <module>
    print(compile_sent(sentence))
  File "/Users/me/PycharmProjects/UDParser/Rough_Work.py", line 99, in compile_sent
    sent_list = sent_list.serialize()
AttributeError: 'list' object has no attribute 'serialize'

The problem is that I'm trying to create a normal list and run the serialize() method on that. I don't know how to create the type of TokenList that is created by the library when the parse() function is run on string in the CoNLL-U file format.

When you try to print that type of list you get the following output:

data = """
# text = The big dog
1   The     the    DET    _    Definite=Def|PronType=Art   _   _   _   _
2   big     big    ADJ    _    Degree=Pos                  _   _   _   _
3   dog     dog    NOUN   _    Number=Sing                 _   _   _   _

"""

sentences = data.parse()
sentence = sentences[0]
print(sentence)

TokenList<The, quick, brown, fox, jumps, over, the, lazy, dog, .>

Running the serialize() method on this type of list will turn it back into a CoNLL-U format string like data in the example above. However, it breaks when you try to run it on a normal python list.

How can I create a TokenList like this instead of a normal python list object?

Answer 1

Change your sent_list from a normal list to a TokenList .

from conllu import TokenList
from collections import OrderedDict

def compile_sent(sent):
    sent_list = TokenList()
    # ... etc ...

You can view the functions on TokenList by using help(TokenList) in a REPL.