提取后使用潜在Dirichlet分配的变换方法时出错

Question

I've trained a Latent Dirichlet Allocation model using sklearn. When I unpickle it, I then use countVectorizer to transform the document and then transform this instance using LDA in order to get the topic distribution but I'm getting the following error:

AttributeError: module '__main__' has no attribute 'tokenize'

Here is my code:

lda = joblib.load('lda_good.pkl')
#LDA trained model

tf_vect = joblib.load('tf_vectorizer_.pkl')
#vectorizer


texts = readContent('doc_name.pdf')

new_doc = tf_vect.transform(texts)

print(new_doc)

print(lda.transform(new_doc))

The thing is that the countVectorizer object unipickled is working fine and I can use .transform but when I try to then .transform using LDA's attribute it seems to refer to the tokenize function from the countvectorizer... The function tokenize is defined ahead of the code, but I can't understant what does tokenize has to do with the transform method of Latent dirichlet Allocation. A weird thing is that all this code is working fine in jupyter notebook but not when i run it as a script..

All the code is in a single file. The model was trained using jupyter notebook and now I was trying to use the model within a script.

Here is the traceback:

Traceback (most recent call last):
File "<string>", line 1, in <module>
Traceback (most recent call last):
File     "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\spawn.py", line 106, in spawn_main
File "<string>", line 1, in <module>
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\spawn.py", line 106, in spawn_main
exitcode = _main(fd)
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\spawn.py", line 115, in _main
exitcode = _main(fd)
prepare(preparation_data)
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\spawn.py", line 115, in _main
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\spawn.py", line 226, in prepare
prepare(preparation_data)
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\spawn.py", line 226, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\spawn.py", line 278, in _fixup_main_from_path
Traceback (most recent call last):
Traceback (most recent call last):
_fixup_main_from_path(data['init_main_from_path'])
File "<string>", line 1, in <module>
File "<string>", line 1, in <module>
run_name="__mp_main__")
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\spawn.py", line 278, in _fixup_main_from_path
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\runpy.py", line 254, in run_path
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\spawn.py", line 106, in spawn_main
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\spawn.py", line 106, in spawn_main
run_name="__mp_main__")
pkg_name=pkg_name, script_name=fname)
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\runpy.py", line 254, in run_path
exitcode = _main(fd)
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\runpy.py", line 96, in _run_module_code
exitcode = _main(fd)
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\spawn.py", line 115, in _main
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\spawn.py", line 115, in _main
pkg_name=pkg_name, script_name=fname)
prepare(preparation_data)
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\spawn.py", line 226, in prepare
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
_fixup_main_from_path(data['init_main_from_path'])
prepare(preparation_data)
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\runpy.py", line 85, in _run_code
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\spawn.py", line 278, in _fixup_main_from_path
mod_name, mod_spec, pkg_name, script_name)
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\spawn.py", line 226, in prepare
exec(code, run_globals)
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\runpy.py", line 85, in _run_code
File "C:\Users\eduard.bermejo\Documents\Machine learning\gestió documental\POC\program_POC.py", line 160, in <module>
run_name="__mp_main__")
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\runpy.py", line 254, in run_path
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\spawn.py", line 278, in _fixup_main_from_path
exec(code, run_globals)
tf_vect = joblib.load('tf_vectorizer_.pkl')
pkg_name=pkg_name, script_name=fname)
File "C:\Users\eduard.bermejo\Documents\Machine learning\gestió    documental\POC\program_POC.py", line 160, in <module>
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\site-  packages\sklearn\externals\joblib\numpy_pickle.py", line 459, in load
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\runpy.py",  line 96, in _run_module_code
run_name="__mp_main__")
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\runpy.py", line 254, in run_path
tf_vect = joblib.load('tf_vectorizer_.pkl')
obj = unpickler.load()
mod_name, mod_spec, pkg_name, script_name)
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\site-packages\sklearn\externals\joblib\numpy_pickle.py", line 459, in load
File  "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\pickle.py", line 1039, in load
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\runpy.py", line 85, in _run_code
pkg_name=pkg_name, script_name=fname)
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\runpy.py", line 96, in _run_module_code
exec(code, run_globals)
File "C:\Users\eduard.bermejo\Documents\Machine learning\gestió documental\POC\program_POC.py", line 160, in <module>
obj = unpickler.load()
mod_name, mod_spec, pkg_name, script_name)
dispatch[key[0]](self)
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\pickle.py", line 1039, in load
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\runpy.py", line 85, in _run_code
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\pickle.py", line 1334, in load_global
tf_vect = joblib.load('tf_vectorizer_.pkl')
exec(code, run_globals)
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\site- packages\sklearn\externals\joblib\numpy_pickle.py", line 459, in load
File "C:\Users\eduard.bermejo\Documents\Machine learning\gestió documental\POC\program_POC.py", line 160, in <module>
dispatch[key[0]](self)
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\pickle.py", line 1334, in load_global
tf_vect = joblib.load('tf_vectorizer_.pkl')
obj = unpickler.load()
klass = self.find_class(module, name)
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\site-packages\sklearn\externals\joblib\numpy_pickle.py", line 459, in load
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\pickle.py", line 1039, in load
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\pickle.py", line 1388, in find_class
klass = self.find_class(module, name)
obj = unpickler.load()
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\pickle.py", line 1388, in find_class
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\pickle.py", line 1039, in load
dispatch[key[0]](self)
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\pickle.py", line 1334, in load_global
return getattr(sys.modules[module], name)
AttributeError: module '__main__' has no attribute 'tokenize'
dispatch[key[0]](self)
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\pickle.py", line 1334, in load_global
klass = self.find_class(module, name)
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\pickle.py", line 1388, in find_class
return getattr(sys.modules[module], name)
AttributeError: module '__main__' has no attribute 'tokenize'
klass = self.find_class(module, name)
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\pickle.py", line 1388, in find_class
return getattr(sys.modules[module], name)
AttributeError: module '__main__' has no attribute 'tokenize'
return getattr(sys.modules[module], name)
AttributeError: module '__main__' has no attribute 'tokenize'
Traceback (most recent call last):
File "<string>", line 1, in <module>
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\spawn.py", line 106, in spawn_main
File "C:\Users\eduard.bermejo\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\spawn.py", line 106, in spawn_main
exitcode = _main(fd)
exitcode = _main(fd)

It actually continues but I guess this will be enough because it starts some kind of loop.

Let me know if further information is needed.

Thanks in advance

Answer 1

Looking through similar questions on SO it shows there are pickling and unpickling issues. The code that you use to do joblib.dump is in a different directory from what I assume. Instead could you put that in the same directory as this program and re-run the pickler and unpickler again? __main__ is stored for the pickled directory and the unpickler tries to search for it when it runs.