堆栈内存溢出
  简体   繁体   English

如何将参数传递给多处理循环

[英]How to pass down parameters to a multiprocessing loop

 原文  2019-06-18 21:37:13  8   1    python/ multithreading/ parallel-processing

问题描述

I want to do a Hartree-Fock kind of computation. 我想做一个Hartree-Fock计算。 In short, it's a self-consistent matrix convergence problem. 简而言之，这是一个自洽的矩阵收敛问题。 I start from a matrix H(0), from which I construct a Fock(0) term. 我从矩阵H（0）开始，从中构造一个Fock（0）项。 Now I define H(1) = H(0)+Fock(0) as my new matrix. 现在我将H（1）= H（0）+ Fock（0）定义为我的新矩阵。 Then I construct Fock(1) from H(1) and define H(2) = H(0)+Fock(1). 然后我从H（1）构造Fock（1）并定义H（2）= H（0）+ Fock（1）。 I repeat this process until H(n+1) is close enough to H(n). 我重复这个过程直到H（n + 1）足够接近H（n）。

The computation is very time consuming, so I am using multiprocessing. 计算非常耗时，所以我使用多处理。 But I find my code doesn't change Fock term as I hoped. 但我发现我的代码并没有像我希望的那样改变Fock术语。 It always uses Fock(0) so the result doesn't make any sense. 它总是使用Fock（0），所以结果没有任何意义。

A piece of simplified code on the multiprocessing part is the following 多处理部分上的一段简化代码如下 

from numpy import *
import multiprocessing as mp
import itertools

kt = 0.3120
q = array([[0,-1],[sqrt(3)/2,1./2],[-sqrt(3)/2,1./2]])*kt

def hbz(x):
    "A set of k points"
    tL = []
    for i in range(-x,x+1):
        for j in range(-x,x+1):
            k = i*q[1]/x+j*q[2]/x
            if linalg.norm(k) <= kt+10**-5 and -q[1][0]-1e-10<=k[0]<=q[1][0]+1e-10:
                tL.append(k)
    return tL

step = 2
d = {}
def H(k):
    return 4*k
def fock(k):
    return -Hp(k)/3
Hp = lambda k: H(k)

def fill_d(k):
    return str(k), 5*fock(k)
if __name__ == '__main__':
    pool = mp.Pool(min(8,mp.cpu_count()-2))
    for ite in range(3):
        d = dict(pool.map_async(fill_d, [kpt for kpt in hbz(step)]).get())
        Hp = lambda k: H(k)+d[str(k)]
        print (d)
        print ("============================")

The output is 输出是 

{'[ 0.    -0.312]': array([-0.  ,  2.08]), '[-0.13509996 -0.234     ]': array([0.90066642, 1.56      ]), '[-0.27019993 -0.156     ]': array([1.80133284, 1.04      ]), '[ 0.13509996 -0.234     ]': array([-0.90066642,  1.56      ]), '[ 0.    -0.156]': array([-0.  ,  1.04]), '[-0.13509996 -0.078     ]': array([0.90066642, 0.52      ]), '[-0.27019993  0.        ]': array([ 1.80133284, -0.        ]), '[ 0.27019993 -0.156     ]': array([-1.80133284,  1.04      ]), '[ 0.13509996 -0.078     ]': array([-0.90066642,  0.52      ]), '[0. 0.]': array([-0., -0.]), '[-0.13509996  0.078     ]': array([ 0.90066642, -0.52      ]), '[-0.27019993  0.156     ]': array([ 1.80133284, -1.04      ]), '[0.27019993 0.        ]': array([-1.80133284, -0.        ]), '[0.13509996 0.078     ]': array([-0.90066642, -0.52      ]), '[0.    0.156]': array([-0.  , -1.04]), '[-0.13509996  0.234     ]': array([ 0.90066642, -1.56      ]), '[0.27019993 0.156     ]': array([-1.80133284, -1.04      ]), '[0.13509996 0.234     ]': array([-0.90066642, -1.56      ]), '[0.    0.312]': array([-0.  , -2.08])}
============================
{'[ 0.    -0.312]': array([-0.  ,  2.08]), '[-0.13509996 -0.234     ]': array([0.90066642, 1.56      ]), '[-0.27019993 -0.156     ]': array([1.80133284, 1.04      ]), '[ 0.13509996 -0.234     ]': array([-0.90066642,  1.56      ]), '[ 0.    -0.156]': array([-0.  ,  1.04]), '[-0.13509996 -0.078     ]': array([0.90066642, 0.52      ]), '[-0.27019993  0.        ]': array([ 1.80133284, -0.        ]), '[ 0.27019993 -0.156     ]': array([-1.80133284,  1.04      ]), '[ 0.13509996 -0.078     ]': array([-0.90066642,  0.52      ]), '[0. 0.]': array([-0., -0.]), '[-0.13509996  0.078     ]': array([ 0.90066642, -0.52      ]), '[-0.27019993  0.156     ]': array([ 1.80133284, -1.04      ]), '[0.27019993 0.        ]': array([-1.80133284, -0.        ]), '[0.13509996 0.078     ]': array([-0.90066642, -0.52      ]), '[0.    0.156]': array([-0.  , -1.04]), '[-0.13509996  0.234     ]': array([ 0.90066642, -1.56      ]), '[0.27019993 0.156     ]': array([-1.80133284, -1.04      ]), '[0.13509996 0.234     ]': array([-0.90066642, -1.56      ]), '[0.    0.312]': array([-0.  , -2.08])}
============================
{'[ 0.    -0.312]': array([-0.  ,  2.08]), '[-0.13509996 -0.234     ]': array([0.90066642, 1.56      ]), '[-0.27019993 -0.156     ]': array([1.80133284, 1.04      ]), '[ 0.13509996 -0.234     ]': array([-0.90066642,  1.56      ]), '[ 0.    -0.156]': array([-0.  ,  1.04]), '[-0.13509996 -0.078     ]': array([0.90066642, 0.52      ]), '[-0.27019993  0.        ]': array([ 1.80133284, -0.        ]), '[ 0.27019993 -0.156     ]': array([-1.80133284,  1.04      ]), '[ 0.13509996 -0.078     ]': array([-0.90066642,  0.52      ]), '[0. 0.]': array([-0., -0.]), '[-0.13509996  0.078     ]': array([ 0.90066642, -0.52      ]), '[-0.27019993  0.156     ]': array([ 1.80133284, -1.04      ]), '[0.27019993 0.        ]': array([-1.80133284, -0.        ]), '[0.13509996 0.078     ]': array([-0.90066642, -0.52      ]), '[0.    0.156]': array([-0.  , -1.04]), '[-0.13509996  0.234     ]': array([ 0.90066642, -1.56      ]), '[0.27019993 0.156     ]': array([-1.80133284, -1.04      ]), '[0.13509996 0.234     ]': array([-0.90066642, -1.56      ]), '[0.    0.312]': array([-0.  , -2.08])}

One can see the parallelized function fill_d is not changing in different iterations. 可以看到并行化函数fill_d在不同的迭代中没有变化。 Since this example is so simple, I can do it without multiprocessing: 由于这个例子很简单，我可以在没有多处理的情况下完成： 

for ite in range(3):
    for kpt in hbz(step):
        d[str(kpt)] = 5*fock(kpt)
    Hp = lambda k: H(k)+d[str(k)]
    print (d)
    print ("============================")

So I know the expected output is 所以我知道预期的输出是 

{'[ 0.    -0.312]': array([-0.  ,  2.08]), '[-0.13509996 -0.234     ]': array([0.90066642, 1.56      ]), '[-0.27019993 -0.156     ]': array([1.80133284, 1.04      ]), '[ 0.13509996 -0.234     ]': array([-0.90066642,  1.56      ]), '[ 0.    -0.156]': array([-0.  ,  1.04]), '[-0.13509996 -0.078     ]': array([0.90066642, 0.52      ]), '[-0.27019993  0.        ]': array([ 1.80133284, -0.        ]), '[ 0.27019993 -0.156     ]': array([-1.80133284,  1.04      ]), '[ 0.13509996 -0.078     ]': array([-0.90066642,  0.52      ]), '[0. 0.]': array([-0., -0.]), '[-0.13509996  0.078     ]': array([ 0.90066642, -0.52      ]), '[-0.27019993  0.156     ]': array([ 1.80133284, -1.04      ]), '[0.27019993 0.        ]': array([-1.80133284, -0.        ]), '[0.13509996 0.078     ]': array([-0.90066642, -0.52      ]), '[0.    0.156]': array([-0.  , -1.04]), '[-0.13509996  0.234     ]': array([ 0.90066642, -1.56      ]), '[0.27019993 0.156     ]': array([-1.80133284, -1.04      ]), '[0.13509996 0.234     ]': array([-0.90066642, -1.56      ]), '[0.    0.312]': array([-0.  , -2.08])}
============================
{'[ 0.    -0.312]': array([-0.        , -1.38666667]), '[-0.13509996 -0.234     ]': array([-0.60044428, -1.04      ]), '[-0.27019993 -0.156     ]': array([-1.20088856, -0.69333333]), '[ 0.13509996 -0.234     ]': array([ 0.60044428, -1.04      ]), '[ 0.    -0.156]': array([-0.        , -0.69333333]), '[-0.13509996 -0.078     ]': array([-0.60044428, -0.34666667]), '[-0.27019993  0.        ]': array([-1.20088856, -0.        ]), '[ 0.27019993 -0.156     ]': array([ 1.20088856, -0.69333333]), '[ 0.13509996 -0.078     ]': array([ 0.60044428, -0.34666667]), '[0. 0.]': array([-0., -0.]), '[-0.13509996  0.078     ]': array([-0.60044428,  0.34666667]), '[-0.27019993  0.156     ]': array([-1.20088856,  0.69333333]), '[0.27019993 0.        ]': array([ 1.20088856, -0.        ]), '[0.13509996 0.078     ]': array([0.60044428, 0.34666667]), '[0.    0.156]': array([-0.        ,  0.69333333]), '[-0.13509996  0.234     ]': array([-0.60044428,  1.04      ]), '[0.27019993 0.156     ]': array([1.20088856, 0.69333333]), '[0.13509996 0.234     ]': array([0.60044428, 1.04      ]), '[0.    0.312]': array([-0.        ,  1.38666667])}
============================
{'[ 0.    -0.312]': array([-0.        ,  4.39111111]), '[-0.13509996 -0.234     ]': array([1.90140689, 3.29333333]), '[-0.27019993 -0.156     ]': array([3.80281377, 2.19555556]), '[ 0.13509996 -0.234     ]': array([-1.90140689,  3.29333333]), '[ 0.    -0.156]': array([-0.        ,  2.19555556]), '[-0.13509996 -0.078     ]': array([1.90140689, 1.09777778]), '[-0.27019993  0.        ]': array([ 3.80281377, -0.        ]), '[ 0.27019993 -0.156     ]': array([-3.80281377,  2.19555556]), '[ 0.13509996 -0.078     ]': array([-1.90140689,  1.09777778]), '[0. 0.]': array([-0., -0.]), '[-0.13509996  0.078     ]': array([ 1.90140689, -1.09777778]), '[-0.27019993  0.156     ]': array([ 3.80281377, -2.19555556]), '[0.27019993 0.        ]': array([-3.80281377, -0.        ]), '[0.13509996 0.078     ]': array([-1.90140689, -1.09777778]), '[0.    0.156]': array([-0.        , -2.19555556]), '[-0.13509996  0.234     ]': array([ 1.90140689, -3.29333333]), '[0.27019993 0.156     ]': array([-3.80281377, -2.19555556]), '[0.13509996 0.234     ]': array([-1.90140689, -3.29333333]), '[0.    0.312]': array([-0.        , -4.39111111])}

How can I modify the multiprocessing code to compute the Fock part according to the H matrix at the iteration? 如何根据迭代时的H矩阵修改多处理代码来计算Fock部分？

1 个解决方案

解决方案1
 0 已采纳  2019-06-20 16:44:10

So I found a way myself. 所以我自己找到了一种方法。 For those who might run into the same problem, I post my solution here. 对于那些可能遇到同样问题的人，我在这里发布我的解决方案。 Basically, there is a similar function in Pool called starmap that accommodates more parameters. 基本上， Pool中有一个类似的函数叫做starmap ，它可以容纳更多的参数。 And I adjusted the order of things a bit. 我调整了一些事情的顺序。 

def fock(k, n):
    return -Hp(k,n)/3
def Hp(k,n):
    if n==0: return H(k)
    else: return H(k)+d[n-1][str(k)] 
d={}
def fill_d(k, n):
    return str(k), 5*fock(k,n)
for ite in range(3):
    if __name__ == '__main__':
        pool = mp.Pool(min(8,mp.cpu_count()-2))    
        d[ite] = dict(pool.starmap_async(fill_d, [(kpt,ite) for kpt in hbz(step)]).get())
        print (d[ite])
        print ("============================")
        pool.close()

暂无
暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 如何将循环转换为多处理？ - How to translate a loop to multiprocessing? 如何通过 pool.imap() function 传递数据以外的参数以在 python 中进行多处理？ - how to pass parameters other than data through pool.imap() function for multiprocessing in python? Python 多处理：如何传递对象？ - Python Multiprocessing : How to pass objects? 如何在for循环中对代码进行多处理/并行化 - How to multiprocessing / parallelizing the code in for loop 如何在for循环中使用多重处理-Python - How to use multiprocessing in a for loop - python 如何退出Python多处理循环？ - How to exit a loop of Python Multiprocessing? 如何使用多参数对for循环进行多处理 - how to multiprocessing a for loop with multi arguments 如何在 for 循环中使用多处理 - How to use multiprocessing within for loop 如何将 while 循环转换为 function 或过程并传入新参数 python - how to convert while loop into a function or procedure and pass in new parameters python 是否可以将默认参数传递给 for 循环？ - Is it possible to pass default parameters to a for loop?
相关标签
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM