从 pickle 文件加载 spacy model 时出错

Question

从 pickle 文件加载 spacy NER model 时出现以下错误。

self.model = pickle.load(open(model_path, 'rb')) 

Traceback (most recent call last):   
File "C:\Program Files\JetBrains\PyCharm Community Edition 2022.2.3\plugins\python-ce\helpers\pydev\pydevd.py", line 1496, in _exec     
pydev_imports.execfile(file, globals, locals)  # execute the script   File "C:\Program Files\JetBrains\PyCharm Community Edition 2022.2.3\plugins\python-ce\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile     
exec(compile(contents+"\n", file, 'exec'), glob, loc)   File 
"C:\Projects\pythonworkspace\invoice_processing_prototype\invoice_data_extractor_notebook.py", line 101, in 
<module>     
extractor = InvoiceDataExtractor(model_dir_path, input_file_paths[0], config_path)   File 
"C:\Projects\pythonworkspace\invoice_processing_prototype\invoicedataextractor.py", line 27, in 
__init__ self.spatial_extractor = SpatialExtractor(model_dir_path, config_path)   
File "C:\Projects\pythonworkspace\invoice_processing_prototype\spatialextractor.py", line 54, in __init__     
self.inv_date = Model(f{self.model_dir_path}\\invoice_date_with_corrected_training_data_and_line_seperator_21_07_2022.pkl",   File 
"C:\Projects\pythonworkspace\invoice_processing_prototype\spatialextractor.py", line 34, in __init__     

self.model = pickle.load(open(model_path, 'rb'))   
File "stringsource", line 6, in spacy.pipeline.trainable_pipe.__pyx_unpickle_TrainablePipe _pickle.PickleError: 
Incompatible checksums (0x417ddeb vs (0x61fbab5, 0x27e6ee8, 0xbe56bc9) = (cfg, model, name, scorer, vocab))

运行以下代码行时出现此错误：

self.model = pickle.load(open(model_path, 'rb'))

我已经使用 spacy 版本 3.1.2 训练了 NER model，最近我将 spacy 升级到了最新的 3.4。 该错误可能是因为某些版本不兼容。 如果是这种情况，有人可以确认是否可以加载在 spacy 版本“3.1.2”上训练的 spacy NER model 可以在升级的 spacy“3.4”上加载

环境

操作系统：Windows 10

Python 使用的版本：3.10

训练时使用的 spaCy 版本：3.1.2

预测时使用的 spaCy 版本：3.4

Answer 1

我找到了解决此错误的一种方法。 在spacy 3.1.2上，我加载了 pickle model 并使用保存

model = pickle.load(open(filepath, 'rb'))    
model.to_disc('my_model')

然后我将 spacy 版本更新到最新的spacy 3.4.4并这次重新加载了“my_model”。

model = spacy.load("my_model")
pickle.dump(model, open(filepath, 'wb'))  

在那种情况下，它对我有用而没有错误。