使用 spaCy 训练 NER 时损失平稳

Question

I'm training (from scratch) a new set of entities and doing exactly as described in spaCy tutorial , however, my loss is plateauing and large number of epochs does not help.

My data:

9 different entities, 15000 training data (sentences). The loss after 20 epochs:

Loaded model 'en'
Losses {'ner': 25461.3508122763}
Losses {'ner': 17003.450728844182}
Losses {'ner': 15725.198527784352}
Losses {'ner': 15315.754479839785}
Losses {'ner': 14980.468680851985}
Losses {'ner': 14716.52629194191}
Losses {'ner': 14346.623731715972}
Losses {'ner': 14463.972966984807}
Losses {'ner': 14195.106732198006}
Losses {'ner': 14058.390174787504}
Losses {'ner': 13875.850727875884}
Losses {'ner': 13859.096326599261}
Losses {'ner': 13614.887464660655}
Losses {'ner': 13512.779816124807}
Losses {'ner': 13388.69595626908}
Losses {'ner': 13496.388241585315}
Losses {'ner': 13530.602194116611}
Losses {'ner': 13245.709490846923}
Losses {'ner': 13219.483523900466}
Losses {'ner': 13189.088232180386}

Question 1:

What is the best way to organise the training data if there are several entities within a single sentence? Should I combine all entities in a list or it is better to train with a single entity?

For example:

("Horses and dogs are too tall and they pretend to care about your feelings", {'entities': [(0, 6, 'ANIMAL'), (11, 15, 'ANIMAL')]})

or it is better to split:

("Horses and dogs are too tall and they pretend to care about your feelings", {'entities': [(0, 6, 'ANIMAL')]}),

("Horses and dogs are too tall and they pretend to care about your feelings", {'entities': [(11, 15, 'ANIMAL')]})

Question 2:

Should I include empty sentences too (with no entities)?

("The new electric cars is great!", {'entities': []})

Apparently, the model predicts not too bad (f1~0.7), however I am wondering what are the best practices to fine tune the model (apart from using the Prodigy on top of this trained model).

Answer 1

spaCy and Prodigy expect different forms of training data: spaCy expects a "gold" annotation, in which every entity is labeled. That annotation format is described in the spaCy docs . If you're just training an NER model, you can simply omit the dependency and POS keys from the dictionary. Training in this way make sense: at prediction time, the model will need to produce entity labels for every word it sees.

Prodigy, in contrast, can accept labeled examples that just have a single span with a proposed entity label, plus a human decision of whether that span is an instance of the entity label or not. This is a little trickier for training, since the model just won't know whether the other words in the sentence are or are not an entity.

My hunch is that the model will work better if you consolidate all the entities in a sentence into one training example (Question 1). This gives the model more information about the sentence and allows it to learn the relationship between different entities in text. (Think, for example of a phrase "she visited X and Y". If X is a place, Y is almost certainly a place. If X is a person, Y is also likely to be). This would be something pretty easy and interesting to check empirically, though.

In regard to question 2, including sentences with no entities should be very helpful for the model.

Side note: when I'm training NER models, the performance usually plateaus after about 20 epochs, and an F1 of 0.7 isn't too bad, so what you're finding sounds about right.