[英]Is it possible to serialize tasklet code (not just exec state) using SPickle without doing a RPC?
Trying to use stackless python (2.7.2) with SPickle to send a test method over celery for execution on a different machine. 尝试通过SPickle使用无堆栈python(2.7.2)在celery上发送测试方法以在另一台计算机上执行。 I would like the test method (code) to be included with the pickle and not forced to exist on the executing machines python path.
我希望测试方法(代码)包含在泡菜中,并且不被迫存在于执行机器的python路径中。
Been referencing following presentation: https://ep2012.europython.eu/conference/talks/advanced-pickling-with-stackless-python-and-spickle 一直在参考以下演示文稿: https : //ep2012.europython.eu/conference/talks/advanced-pickling-with-stackless-python-and-spickle
Trying to use the technique shown in the checkpointing slide 11. The RPC example doesn't seem right given that we are using celery: 尝试使用检查点幻灯片11中显示的技术。考虑到我们正在使用celery,RPC示例似乎并不正确:
Client code: 客户代码:
from stackless import run, schedule, tasklet
from sPickle import SPickleTools
def test_method():
print "hello from test method"
tasks = []
test_tasklet = tasklet(test_method)()
tasks.append(test_tasklet)
pt = SPickleTools(serializeableModules=['__test_method__'])
pickled_task = pt.dumps(tasks)
Server code: 服务器代码:
pt = sPickle.SPickleTools()
unpickledTasks = pt.loads(pickled_task)
Results in: 结果是:
[2012-03-09 14:24:59,104: ERROR/MainProcess] Task
celery_tasks.test_exec_method[8f462bd6-7952-4aa1-9adc-d84ee4a51ea6] raised exception:
AttributeError("'module'
object has no attribute 'test_method'",)
Traceback (most recent call last):
File "c:\Python27\lib\site-packages\celery\execute\trace.py", line 153, in trace_task
R = retval = task(*args, **kwargs)
File "c:\Python27\celery_tasks.py", line 16, in test_exec_method
unpickledTasks = pt.loads(pickled_task)
File "c:\Python27\lib\site-packages\sPickle\_sPickle.py", line 946, in loads
return unpickler.load()
AttributeError: 'module' object has no attribute 'test_method'
Any suggestions on what I am doing incorrect or if this is even possible? 关于我在做什么不正确或什至有可能的任何建议?
Alternative suggestions for doing dynamic module loading in a celeryd would also be good (as an alternative for using sPickle). 在celeryd中进行动态模块加载的替代建议也将是不错的(作为使用sPickle的替代方法)。 I have experimented with doing:
我已经尝试过这样做:
py_mod = imp.load_source(module_name,'some script path')
sys.modules.setdefault(module_name,py_mod)
but the dynamically loaded module does not seem to persist through different calls to celeryd, ie different remote calls. 但是动态加载的模块似乎不会通过对celeryd的不同调用(即不同的远程调用)而保持不变。
You must define test_method
within its own module. 您必须在其自己的模块中定义
test_method
。 Currently sPickle detects whether test_method
is defined in a module that can be imported. 当前,sPickle检测是否在可以导入的模块中定义了
test_method
。 An alternative way is to set the __module__
attribute of the function to None
. 另一种方法是将函数的
__module__
属性设置为None
。
def test_method():
pass
test_method.__module__ = None
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.