sklearn.externals joblib 抛出 ImportError - 服务器上没有名为 joblib.numpy_pickle 的模块

Question

I am struggling with a joblib issue.

I need to push inference application to a remote server, and I also need to load a saved standard scaler, because if I try to fit, i get an out of memory exception. I can't make phisical changes to the server, as it is a sap cloud foundry instance , and I don't see a way to solve it.

Traceback from server:

   2019-08-05T15:34:41.54+0200 [APP/PROC/WEB/0] ERR   File "app.py", line 40, in <module>
   2019-08-05T15:34:41.54+0200 [APP/PROC/WEB/0] ERR     standard_scaler = joblib.load('./datascaler.save')
   2019-08-05T15:34:41.54+0200 [APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python2.7/site-packages/sklearn/externals/joblib/numpy_pickle.py", line 598, in load
   2019-08-05T15:34:41.54+0200 [APP/PROC/WEB/0] ERR     obj = _unpickle(fobj, filename, mmap_mode)
   2019-08-05T15:34:41.54+0200 [APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python2.7/site-packages/sklearn/externals/joblib/numpy_pickle.py", line 526, in _unpickle
   2019-08-05T15:34:41.54+0200 [APP/PROC/WEB/0] ERR     obj = unpickler.load()
   2019-08-05T15:34:41.54+0200 [APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python2.7/pickle.py", line 864, in load
   2019-08-05T15:34:41.54+0200 [APP/PROC/WEB/0] ERR     dispatch[key](self)
   2019-08-05T15:34:41.54+0200 [APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python2.7/pickle.py", line 1096, in load_global
   2019-08-05T15:34:41.54+0200 [APP/PROC/WEB/0] ERR     klass = self.find_class(module, name)
   2019-08-05T15:34:41.54+0200 [APP/PROC/WEB/0] ERR   File "/home/vcap/deps/0/python/lib/python2.7/pickle.py", line 1130, in find_class
   2019-08-05T15:34:41.54+0200 [APP/PROC/WEB/0] ERR     __import__(module)
   2019-08-05T15:34:41.54+0200 [APP/PROC/WEB/0] ERR ImportError: No module named joblib.numpy_pickle

From what I have seen on similar posts, joblib version conflict/mismatch might be a cause, but I can't really do anything with this, since its not my local machine that has this error.

Is there any other way to save the scaler, or to solve this conflict?

Answer 1

i also encount this problem. there may be two reasons.

1. the mode saved with sklearn.externals.joblib while loading with joblib . (sklearn.externals.joblib and joblib are different libs)

2. both sklearn.externals.joblib are used in save and load stage. however, they are different scikit-learn versions. such as, in save stage scikit-learn==0.19.1 while in load stage scikit-learn==0.21.x .

the best way to repair it is use joblib with same versions. sklearn.externals.joblib will be removed in future version.