将 NER 训练数据转换为 Spacy 训练数据格式

Question

I am creating an Indonesian NER model using Spacy. I'm using training data from https://raw.githubusercontent.com/yohanesgultom/nlp-experiments/master/data/ner/training_data.txt

Above training data using this Tag format:

Sementara itu Pengamat Pasar Modal <ENAMEX TYPE="PERSON">Dandossi Matram</ENAMEX> mengatakan,

I wanted to convert this training data to Spacy format that is:

[('Sementara itu Pengamat Pasar Modal Dandossi Matram mengatakan,',{"entities:"([35, 51, 'PERSON'])})]

I'm still new to Python library, any idea how to convert the train data? Or any idea to use which library?

Thank you.

Answer 1

For simple XML-type annotations you can use BeautifulSoup. Here's an example with slightly simpler markup:

from bs4 import BeautifulSoup

raw = "I went to <PLACE>Tokyo 3</PLACE> last year."
soup = BeautifulSoup(raw, features="html.parser")

out = ""
tags = []
idx = 0
for el in soup:
    text = el
    if hasattr(el, "text"):
        # it's a tag, save it
        text = el.text
        start = idx
        end = idx + len(el.text)
        tags.append( (el.name, start, end) )

    out += text
    idx += len(text)

print(out)
for tag in tags:
    print(tag[0], out[tag[1]:tag[2]], sep="\t")

Once you have the character spans like this example code gives, getting the spaCy format data is straightforward.