[英]Cython Pickling in Package “not found as” Error
I'm having trouble pickling a Cython class, but only when it's defined inside a package. 我在挑选一个Cython类时遇到了麻烦,但只有当它在一个包中定义时才会出现问题。 This problem was noted previously online , but they didn't state how it was resolved.
之前在线注意到了这个问题,但他们没有说明它是如何解决的。 There are two components here: the Cython pickling using a
__reduce__
method and a package error. 这里有两个组件:使用
__reduce__
方法的Cython酸洗和包错误。
I'll first show how it works without the package part. 我将首先展示它如何在没有包装部件的情况下工作。 This example works correctly.
此示例正常工作。
My Cython file is reudce.pyx
: 我的Cython文件是
reudce.pyx
:
cdef class Foo(object):
cdef int n
def __init__(self, n):
self.n = n
def __reduce__(self):
return Foo, (self.n,)
This may be compiled with a setup.py
: 这可以使用
setup.py
编译:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
cmdclass = {'build_ext': build_ext},
ext_modules = [Extension("reduce", ["reduce.pyx"])]
)
by executing python setup.py build && cp build/lib*/reduce.so .
通过执行
python setup.py build && cp build/lib*/reduce.so .
The test script is called test_reduce.py
and is: 测试脚本名为
test_reduce.py
,它是:
import reduce
import pickle
f = reduce.Foo(4)
print pickle.dumps(f)
Executing python test_reduce.py
works fine. 执行
python test_reduce.py
工作正常。
However, once the reduce.pyx
is put into a package, there is an error. 但是,一旦将
reduce.pyx
放入包中,就会出错。
To reproduce this, first create a package called bar
. 要重现这一点,首先要创建一个名为
bar
的包。
mkdir bar
mv reduce.so bar
echo "from reduce import Foo" > bar/__init__.py
Change the test_reduce.py
file to be: 将
test_reduce.py
文件更改为:
import bar
import pickle
f = bar.Foo(4)
print pickle.dumps(f)
Running python test_reduce.py
gives the following error: 运行
python test_reduce.py
会出现以下错误:
File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.7/pickle.py", line 748, in save_global
(obj, module, name))
pickle.PicklingError: Can't pickle <type 'reduce.Foo'>: it's not found as reduce.Foo
There is a catch of errors which are all turned into a PicklingError in pickle.py
After looking at that code, the specific error which is occuring is: 有
pickle.py
错误在pickle.py
变成了pickle.py
。查看该代码后,发生的具体错误是:
ImportError: No module named reduce
To check that there is not some kind of scope or other issue, if I run the steps which the pickle module should execute, everything works: 要检查是否存在某种范围或其他问题,如果我运行pickle模块应执行的步骤,一切正常:
f = bar.Foo(4)
call, args = f.__reduce__()
print call(*args)
So what's going on here?! 那么这里发生了什么?!
The problem was in the build script. 问题出在构建脚本中。 The
Pickle
module uses the __module__
attribute of a function/class for pickling. Pickle
模块使用函数/类的__module__
属性进行酸洗。 That __module__
attribute comes from the first argument to the Extension()
constructor in the setup.py
script. __module__
属性来自setup.py
脚本中Extension()
构造函数的第一个参数。 Since I defined my constructor to be Extension('reduce', ['reduce.pyx'])
, the __module__
attribute is reduce
. 由于我将构造函数定义为
Extension('reduce', ['reduce.pyx'])
,因此__module__
属性为reduce
。 It should be bar/reduce
though since it's now in a package. 它应该是
bar/reduce
因为它现在在一个包中。
Making setup.py
look like: 使
setup.py
看起来像:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
cmdclass = {'build_ext': build_ext},
ext_modules = [Extension('bar/reduce', ['reduce.pyx'])]
)
solves the problem. 解决了这个问题。
EDIT 编辑
maybe something like this is possible: Foo.__module__ = 'bar.reduce'
in bar/__init__.py
也许这样的事情是可能的:
Foo.__module__ = 'bar.reduce'
在bar/__init__.py
Or you can use the module copyreg
. 或者您可以使用模块
copyreg
。 Here are some snippets from the code of copyreg: 以下是copyreg代码中的一些片段:
"""Helper to provide extensibility for pickle.
This is only useful to add pickle support for extension types defined in
C, not for instances of user-defined classes.
"""
def pickle(ob_type, pickle_function, constructor_ob=None):
# ...
# Example: provide pickling support for complex numbers.
try:
complex
except NameError:
pass
else:
def pickle_complex(c):
return complex, (c.real, c.imag)
pickle(complex, pickle_complex, complex)
OLD VERSION 旧版
Pickle does this: Pickle这样做:
try:
__import__(module, level=0)
mod = sys.modules[module]
klass = getattr(mod, name)
except (ImportError, KeyError, AttributeError):
raise PicklingError(
"Can't pickle %r: it's not found as %s.%s" %
(obj, module, name))
Can you try out which line fails? 你能试试哪条线路失败吗?
When pickle piclkes classes and functions like module.function
it reasssures itself that this is the right function meant: 当pickle piclkes类和函数如
module.function
它会重新确认这是正确的函数意味着:
# module.py
def function(): # 1
pass
a = function
def function(): # 2
pass
Function 1 in a
can not be pickled but function 2 can be pickled because it is found in the module under the __name__
it has which is "function". 在功能1
a
不能被酸洗,但因为它的下模块中发现的功能2可以腌__name__
它具有为“功能”。
So in your case pickle does not find the same class Foo
in the reduce
module as is passed as argument. 因此,在您的情况下,pickle在
reduce
模块中找不到与作为参数传递的相同的类Foo
。 The argument Foo
claims to be found in the module reduce
. Foo
声称可以在模块reduce
找到。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.