如何使用流数据训练 spacy 模型？

Question

I have created a spacy model. But I need to retrain it until it reaches it maximum level. I need to train this model and retrain the model using the streaming data. I have seen that we can train some machine learning model using stream data. Is it possible to do the same to NLP models?

Answer 1

You can write a custom corpus reader for your data ( https://spacy.io/api/top-level#corpus-readers ) and use the setting max_epochs = -1 to indicate that the data should be streamed:

[training]
max_epochs = -1

Depending on the type of component and the data, you may need to additionally initialize the labels for the component in the [initialize] block. If you're not streaming, the labels are automatically initialized from the full training corpus, but you can use spacy init labels to generate the labels based on a subset of the data instead and initialize them separately.

More details: https://spacy.io/usage/v3-1#streaming-corpora