如何使用泡菜文件进行 Spacy NER model 培训？

Question

I have data for fine-tuning in this format:

[[(('Kaweah', 'NNP'), 'O'),
  (('Delta', 'NNP'), 'O'),
  (('Mental', 'NNP'), 'O'),
  (('Health', 'NNP'), 'O'),
  (('Hospital', 'NNP'), 'O'),
  (('D/p', 'NNP'), 'O'),
  (('Aph', 'NNP'), 'O'),
  (('is', 'VBZ'), 'O'),
  (('located', 'VBN'), 'O'),
  (('at', 'IN'), 'O'),
  (('1100', 'CD'), 'B-GPE'),
  (('SO', 'NNP'), 'I-GPE'),
  (('.', '.'), 'I-GPE'),
  (('AKERS', 'NNP'), 'I-GPE'),
  (('STREET', 'NNP'), 'I-GPE')],
 [(('CHARLTON', 'NNP'), 'O'),
  (('MEMORIAL', 'NNP'), 'O'),
  (('HOSPITAL', 'NNP'), 'O'),
  (('is', 'VBZ'), 'O'),
  (('located', 'VBN'), 'O'),
  (('at', 'IN'), 'O'),
  (('2449', 'CD'), 'B-GPE'),
  (('THIRD', 'NNP'), 'I-GPE'),
  (('STREET', 'NNP'), 'I-GPE'),
  ((',', ','), 'I-GPE'),
  (('GA', 'NNP'), 'I-GPE')]]

But spacy training format is looking like this:

TRAIN_DATA =[ ("Pizza is a common fast food.", {"entities": [(0, 5, "FOOD")]}),
              ("Pasta is an italian recipe", {"entities": [(0, 5, "FOOD")]}) ]

What should I do to convert my pickle file to the spacy format?

Answer 1

You can just take each token and concatenate whitespace on them to get the text input and then calculate the start and stop for the spans of each entity label.

but you will be creating artificial training data since you don't know what whitespace was used around each token. That information is lost in CoNLL format.

Therefore, the NER model you train will not be robust since it doesn't know how to label tokens as it will learn to always expect the whitespace concatenation of your choice .

Basically, can't be done , you lost information in CoNLL format which cant be recuperated.