[英]Python multiprocessing: How to close the multiprocessing pool on exception
[英]Can't we initialize and close python multiprocessing pool multiple times?
它為fileNum=1機器人而不是為fileNum=2累積
我們不能多次初始化和關閉pool
嗎?
import multiprocessing
import numpy as np
def prepare_data_fill_arrays(fileNum,simNum,chrLong):
arrayList=[]
array1=np.ones((10,))*simNum*fileNum
array2=np.ones((10,))*simNum*fileNum
arrayList.append(simNum)
arrayList.append(chrLong)
arrayList.append(array1)
arrayList.append(array2)
return arrayList
if __name__ == '__main__':
numofFiles=2
numofSimulations = 10
chromNamesList = ['chr1', 'chr2', 'chr3', 'chr4', 'chr5', 'chr6', 'chr7', 'chrX', 'chr8', 'chr9', 'chr10', 'chr11',
'chr12', 'chr13', 'chr14', 'chr15', 'chr16', 'chr17', 'chr18', 'chr20', 'chrY', 'chr19', 'chr22',
'chr21', 'chrM']
sim_nums = range(0, numofSimulations + 1)
sim_num_chr_tuples = ((sim_num, chrLong) for sim_num in sim_nums for chrLong in chromNamesList)
files=range(1, numofFiles+1)
for fileNum in files:
jobs = []
accumulatedArray1 = np.zeros((numofSimulations, 10))
accumulatedArray2 = np.zeros((numofSimulations, 10))
def accumulateArray(arrayList):
try:
simNum = arrayList[0]
chrLong = arrayList[1]
array1 = arrayList[2]
array2 = arrayList[3]
accumulatedArray1[simNum - 1] += array1
accumulatedArray2[simNum - 1] += array2
# print('ACCUMULATION simNum:%d chrLong:%s' % (simNum, chrLong))
except Exception as e:
print("Exception: %s" % (e))
numofProcesses = multiprocessing.cpu_count()
pool = multiprocessing.Pool(processes=numofProcesses)
for simNum, chrLong in sim_num_chr_tuples:
jobs.append(pool.apply_async(prepare_data_fill_arrays,
args=(fileNum,simNum,chrLong,),
callback=accumulateArray))
for job in jobs:
job.get()
pool.close()
pool.join()
print('###################################')
print('filenum:%d' %(fileNum))
print("accumulatedArray1:%s" %(accumulatedArray1))
print("accumulatedArray2:%s" %(accumulatedArray2))
嘗試改變這個
sim_num_chr_tuples = ((sim_num, chrLong) for sim_num in sim_nums for chrLong in chromNamesList)
到,例如,
sim_num_chr_tuples = [(sim_num, chrLong) for sim_num in sim_nums for chrLong in chromNamesList]
也就是說,你想多次迭代sim_num_chr_tuples
,但原來的拼寫是一個生成器表達式,在你第一次迭代之后就用完了。 第二遍,
for simNum, chrLong in sim_num_chr_tuples:
無事可做。 而是具體化為一個列表,然后您可以對其進行多次迭代。
這與多處理無關,順便說一句。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.