从 pickle 文件加载 spacy model 时出错

Question

I am getting the following error while loading spacy NER model from the pickle file.

self.model = pickle.load(open(model_path, 'rb')) 

Traceback (most recent call last):   
File "C:\Program Files\JetBrains\PyCharm Community Edition 2022.2.3\plugins\python-ce\helpers\pydev\pydevd.py", line 1496, in _exec     
pydev_imports.execfile(file, globals, locals)  # execute the script   File "C:\Program Files\JetBrains\PyCharm Community Edition 2022.2.3\plugins\python-ce\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile     
exec(compile(contents+"\n", file, 'exec'), glob, loc)   File 
"C:\Projects\pythonworkspace\invoice_processing_prototype\invoice_data_extractor_notebook.py", line 101, in 
<module>     
extractor = InvoiceDataExtractor(model_dir_path, input_file_paths[0], config_path)   File 
"C:\Projects\pythonworkspace\invoice_processing_prototype\invoicedataextractor.py", line 27, in 
__init__ self.spatial_extractor = SpatialExtractor(model_dir_path, config_path)   
File "C:\Projects\pythonworkspace\invoice_processing_prototype\spatialextractor.py", line 54, in __init__     
self.inv_date = Model(f{self.model_dir_path}\\invoice_date_with_corrected_training_data_and_line_seperator_21_07_2022.pkl",   File 
"C:\Projects\pythonworkspace\invoice_processing_prototype\spatialextractor.py", line 34, in __init__     

self.model = pickle.load(open(model_path, 'rb'))   
File "stringsource", line 6, in spacy.pipeline.trainable_pipe.__pyx_unpickle_TrainablePipe _pickle.PickleError: 
Incompatible checksums (0x417ddeb vs (0x61fbab5, 0x27e6ee8, 0xbe56bc9) = (cfg, model, name, scorer, vocab))

I am getting this error while running the following line of code:

self.model = pickle.load(open(model_path, 'rb'))

I have trained the NER model using the spacy version 3.1.2 and I recently upgraded the spacy to the latest 3.4. The error might be because of some version incompatibilities. If that is the case can someone confirm if is it possible to load spacy NER model trained on spacy version '3.1.2' can be loaded on the upgraded spacy '3.4'

Environment

Operating System: Windows 10

Python Version Used: 3.10

spaCy Version Used while training: 3.1.2

spaCy Version Used while prediction: 3.4

Answer 1

I found one workaround to this error. While on spacy 3.1.2 I have loaded the pickle model and saved it using

model = pickle.load(open(filepath, 'rb'))    
model.to_disc('my_model')

Then I have updated the spacy version to latest spacy 3.4.4 and reloaded the 'my_model' this time.

model = spacy.load("my_model")
pickle.dump(model, open(filepath, 'wb'))  

In that case it worked for me without the error.