I am facing a problem, not that good in coding,I have a tsv file where data looks like this:
lines are separated by a blank line. I have tried using this:
def load_data_spacy(file_path):
''' Converts data from:
word \t label \n word \t label \n \n word \t label
to: sentence, {entities : [(start, end, label), (stard, end, label)]}
'''
file = open(file_path, 'r')
training_data, entities, sentence, unique_labels = [], [], [], []
current_annotation = None
start =0
end = 0 # initialize counter to keep track of start and end characters
for line in file:
line = line.strip("\n").split("\t")
# lines with len > 1 are words
if len(line) > 1:
label = line[1]
if(label != 'O'):
label = line[1] # the .txt is formatted: label \t word, label[0:2] = label_type
#label_type = line[0][0] # beginning of annotations - "B", intermediate - "I"
word = line[0]
sentence.append(word)
start = end
end += (len(word) + 1) # length of the word + trailing space
# lines with len == 1 are breaks between sentences
if len(line) == 1:
if(len(entities) > 0):
sentence = " ".join(sentence)
training_data.append([sentence, {'entities' : entities}])
# reset the counters and temporary lists
end = 0
start = 0
entities, sentence = [], []
file.close()
return training_data, unique_labels
But I am unable to get the required spacy format for NER which should look like this:
[["sentence", {'entities': [(start, end, 'tags')]}]
You don't have to write custom code for this. This is one of the formats spaCy can convert directly using the spacy convert
command. It's the conll/ner
format, so you can just do this:
spacy convert -c ner myfile.tsv out.spacy
Note that as of v3 spaCy doesn't have a specific JSON format that's recommended, you just need to make Docs that look like the output you want. Take a look at the example projects to see code converting various types of data.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.