[英]Passing sympy lambda to multiprocessing.Pool.map
I want to execute a sympy lambda function in parallel.我想并行执行一个 sympy lambda 函数。 I don't know:
我不知道:
lambdify
lambdify
的第一个返回的注释,为什么它会起作用And apparently the markdown preprocessor needs a line of text above the code so this is the code:显然,降价预处理器需要在代码上方一行文本,所以这是代码:
from multiprocessing import Pool
import sympy
from sympy.abc import x
def f(m):
return m.lambdify()(1)
class Mult():
def lambdify(self):
# return sympy.lambdify(x, 2*x, 'numpy')
self._lambdify = sympy.lambdify(x, 2 * x, 'numpy')
return self._lambdify
if __name__ == '__main__':
with Pool() as pool:
m = Mult()
print(pool.map(f, [m]))
print(pool.map(f, [m]))
print(f(m))
print(pool.map(f, [m]))
It prints:它打印:
[2]
[2]
2
PicklingError: Can't pickle <function <lambda> at 0x000000000DF0D048>: attribute lookup <lambda> on numpy failed
(I cut the traceback) (我削减了追溯)
If I uncomment, it works normally:如果我取消注释,它会正常工作:
[2]
[2]
2
[2]
I tested only on Windows and it works exactly the same with 'numexpr' instead of 'numpy'.我只在 Windows 上进行了测试,它与 'numexpr' 而不是 'numpy' 的工作方式完全相同。
The object Mult
has no fields when it is created.对象
Mult
在创建时没有字段。 It can thus be pickled with the stock pickle
library.因此,它可以使用常备
pickle
库进行腌制。 Then, when you call lambdify
, you add a _lambdify
attribute to the object containing a lambda
expression, which cannot be pickled.然后,当您调用
lambdify
,您将_lambdify
属性添加到包含无法腌制的lambda
表达式的对象。 This causes a failure in the map
function这会导致
map
功能失败
This explains why before calling lambdify
you can pickle the object and use Pool.map
and why it fails after the call.这解释了为什么在调用
lambdify
之前您可以pickle 对象并使用Pool.map
以及为什么它在调用后失败。 When you uncomment the line in lambdify
, you do not add the attribute to the class, and the Mult
object can still be pickled after calling lambdify
.当您取消对
lambdify
的行进行lambdify
,您不会将该属性添加到类中,并且在调用lambdify
后仍然可以腌制Mult
对象。
Though I have not fully explored this yet, I just want to put on record that the same example works just fine when using loky instead of multiprocessing:虽然我还没有完全探索这个,但我只是想记录一下,当使用 loky 而不是 multiprocessing 时,同一个例子工作得很好:
from loky import get_reusable_executor
import sympy
from sympy.abc import x
def f(m):
return m.lambdify()(1)
class Mult():
def lambdify(self):
# return sympy.lambdify(x, 2*x, 'numpy')
self._lambdify = sympy.lambdify(x, 2 * x, 'numpy')
return self._lambdify
executor = get_reusable_executor()
m = Mult()
print('pool.map(f, [m])', list(executor.map(f, [m])))
print('pool.map(f, [m])', list(executor.map(f, [m])))
print('f(m)', f(m))
print('pool.map(f, [m])', list(executor.map(f, [m])))
with output带输出
pool.map(f, [m]) [2]
pool.map(f, [m]) [2]
f(m) 2
pool.map(f, [m]) [2]
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.