[英]recursive function calls and queuing in python
I have the sketch of my code as follows: 我的代码如下:
def func1(c):
return a,b
def func2(c,x):
if condition:
a,b = func1(c)
x.append(a,b)
func2(a,x)
func2(b,x)
return x
x = []
y = func2(c, x)
The problem, as you might have figured out from the code, is that I would like func2(b)
to be computed in parallel with func2(a)
whenever condition is true ie before b
is replace by a
new b
from func2(a)
. 正如您可能已经从代码中发现的那样,问题是我希望只要条件为真,即在
b
被func2(a)
a
新b
替换之前,就希望func2(b)
与func2(a)
并行计算。 。 But according to my algorithm, this clearly can not happen due to the new b's. 但是根据我的算法,由于新的b,这显然不会发生。
I do think such a problem might be perfect for parallel computing approach. 我确实认为这样的问题对于并行计算方法可能是完美的。 But, I did not use it before and my knowledge about that is quite limited.
但是,我以前没有使用过它,对此我的知识非常有限。 I did try the suggestion from How to do parallel programming in Python , though.
不过,我确实尝试了“ 如何在Python中进行并行编程”中的建议。 But I got the same result like the sketch above.
但是我得到了与上面的草图相同的结果。
Caveat: Threading might not be parallel enough for you (see https://docs.python.org/2/library/threading.html note on the Global Interpreter Lock
) so you might have to use the multiprocessing
library instead ( https://docs.python.org/2/library/multiprocessing.html ). 警告:线程可能不够并行(请参阅https://docs.python.org/2/library/threading.html有关
Global Interpreter Lock
),因此您可能不得不使用multiprocessing
库( https:/ /docs.python.org/2/library/multiprocessing.html )。
...So I've cheated/been-lazy & used a thread/process neutral term "job". ...所以我作弊/变得懒惰,并使用线程/进程中性术语“工作”。 You'll need to pick either threading or multiprocessing for everywhere that I use "job".
您需要为我使用“作业”的所有位置选择线程处理或多处理。
def func1(c):
return a,b
def func2(c,x):
if condition:
a,b = func1(c)
x.append(a,b)
a_job = None
if (number_active_jobs() >= NUM_CPUS):
# do a and b sequentially
func2(a, x)
else:
a_job = fork_job(func2, a, x)
func2(b,x)
if a_job is not None:
join(a_job)
x = []
func2(c, x)
# all results are now in x (don't need y)
...that will be best if you need a,b pairs to finish together for some reason. ...如果由于某种原因需要a,b对一起完成,那将是最好的。 If you're willing to let the scheduler go nuts, you could "job" them all & then
join
at the end: 如果您愿意放弃调度程序,则可以“全部”使用它们,然后在最后
join
:
def func1(c):
return a,b
def func2(c,x):
if condition:
a,b = func1(c)
x.append(a,b)
if (number_active_jobs() >= NUM_CPUS):
# do a and b sequentially
func2(a, x)
else:
all_jobs.append(fork_job(func2, a, x))
# TODO: the same job-or-sequential for func2(b,x)
all_jobs = []
x = []
func2(c, x)
for j in all_jobs:
join(j)
# all results are now in x (don't need y)
The NUM_CPUS check could be done with threading.activeCount()
instead of a full blown worker threa pool ( python - how to get the numebr of active threads started by specific class? ). NUM_CPUS检查可以通过
threading.activeCount()
而不是完整的工作线程池来完成(python-如何让特定类启动的活动线程数? )。
But with multiprocessing you'd have more work to do with JoinableQueue
and a fixed size Pool
of workers 但是通过多处理,您将需要使用
JoinableQueue
和固定大小的工作Pool
来做更多的工作
From your explanation I have a feeling that it is not that b
gets updated (which is not, as DouglasDD explained), but x
. 从您的解释中,我感觉不是
b
得到更新(正如DouglasDD解释的那样不是),而是x
。 To let both recursive calls to work on a same x
, you need to take some sort of a snapshot of x
. 为了让两个递归调用都可以在同一个
x
,您需要拍摄x
的某种快照。 The simplest way is to pass an index of a newly appended tuple, along the lines of 最简单的方法是沿着的行传递新附加的元组的索引
def func2(c, x, length):
...
x.append(a, b)
func2(a, x, length + 1)
func2(b, x, length + 1)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.