将 .tsv 格式转换为 NER 的 spacy

Question

I am facing a problem, not that good in coding,I have a tsv file where data looks like this:

lines are separated by a blank line. I have tried using this:

def load_data_spacy(file_path):
''' Converts data from:
word \t label \n word \t label \n \n word \t label
to: sentence, {entities : [(start, end, label), (stard, end, label)]}
'''
file = open(file_path, 'r')
training_data, entities, sentence, unique_labels = [], [], [], []
current_annotation = None
start =0
end = 0 # initialize counter to keep track of start and end characters
for line in file:
    line = line.strip("\n").split("\t")
    # lines with len > 1 are words
    if len(line) > 1:
        label = line[1]
        if(label != 'O'):
            label = line[1]     # the .txt is formatted: label \t word, label[0:2] = label_type
        #label_type = line[0][0] # beginning of annotations - "B", intermediate - "I"
        word = line[0]
        sentence.append(word)
        start = end
        end += (len(word) + 1)  # length of the word + trailing space
       # lines with len == 1 are breaks between sentences
    if len(line) == 1:
        if(len(entities) > 0):
            sentence = " ".join(sentence)
            training_data.append([sentence, {'entities' : entities}])
        # reset the counters and temporary lists
        end = 0 
        start = 0
        entities, sentence = [], []
        
file.close()
return training_data, unique_labels

But I am unable to get the required spacy format for NER which should look like this:

[["sentence", {'entities': [(start, end, 'tags')]}]

Answer 1

You don't have to write custom code for this. This is one of the formats spaCy can convert directly using the spacy convert command. It's the conll/ner format, so you can just do this:

spacy convert -c ner myfile.tsv out.spacy

Note that as of v3 spaCy doesn't have a specific JSON format that's recommended, you just need to make Docs that look like the output you want. Take a look at the example projects to see code converting various types of data.