如何使用混淆矩阵计算自定义训练的 spacy ner 模型的整体精度？

Question

I'm trying to evaluate my custom trained Spacy NER model. How to find the overall accuracy with confusion matrix for the model.

I tried evaluating the model with spacy scorer which gives precision, recall and token accuracy with the below reference,

Evaluation in a Spacy NER model

I expect the output in confusion matrix instead of individual precision, recall and token accuracy.

Answer 1

Here is a good read for creating Confusion Matrices for Spacy NER models. It is based on the BILOU format used by Spacy. It is good for small portions of text but when bigger documents are evaluated a Confusion Matrix is hard to read because most pieces of the text are O-labeled.

What you can do is create two lists, one with predicted values per word and one with the true values per word and compare those using the sklearn.metrics.confusion_matrix() function.

from sklearn.metrics import confusion_matrix
y_true = [O,O,O,B-PER,I-PER]
y_pred = [O,O,O,B-PER,O]
confusion_matrix(y_true, y_pred, labels=["O", "B-PER", "I-PER"])

You can also use the plot_confusion_matrix() function from the same library to get a visual output, however this requires scikit-learn 0.23.1 or above and is only usable with scikit-learn classifiers.

As written in this stackoverflow question, this is a way to use the confusion_matrix() from scikit-learn without their plot.

from sklearn.metrics import confusion_matrix

labels = ['business', 'health']
cm = confusion_matrix(y_test, pred, labels)
print(cm)
fig = plt.figure()
ax = fig.add_subplot(111)
cax = ax.matshow(cm)
plt.title('Confusion matrix of the classifier')
fig.colorbar(cax)
ax.set_xticklabels([''] + labels)
ax.set_yticklabels([''] + labels)
plt.xlabel('Predicted')
plt.ylabel('True')
plt.show()