[英]How would I use Dask to perform parallel operations on slices of NumPy arrays?
I have a numpy array of coordinates of size n_slice x 2048 x 3, where n_slice is in the tens of thousands. 我有一个大小为n_slice x 2048 x 3的numpy坐标数组,其中n_slice是数万个。 I want to apply the following operation on each 2048 x 3 slice separately 我想分别在每个2048 x 3切片上应用以下操作
import numpy as np
from scipy.spatial.distance import pdist
# load coor from a binary xyz file, dcd format
n_slice, n_coor, _ = coor.shape
r = np.arange(n_coor)
dist = np.zeros([n_slice, n_coor, n_coor])
# this loop is what I want to parallelize, each slice is completely independent
for i in xrange(n_slice):
dist[i, r[:, None] < r] = pdist(coor[i])
I tried using Dask by making coor
a dask.array
, 我试着用DASK通过使coor
一个dask.array
,
import dask.array as da
dcoor = da.from_array(coor, chunks=(1, 2048, 3))
but simply replacing coor
by dcoor
will not expose the parallelism. 但是简单地用dcoor
替换coor
不会暴露并行性。 I could see setting up parallel threads to run for each slice but how do I leverage Dask to handle the parallelism? 我可以看到设置并行线程为每个切片运行但是如何利用Dask来处理并行性?
Here is the parallel implementation using concurrent.futures
这是使用concurrent.futures
的并行实现
import concurrent.futures
import multiprocessing
n_cpu = multiprocessing.cpu_count()
def get_dist(coor, dist, r):
dist[r[:, None] < r] = pdist(coor)
# load coor from a binary xyz file, dcd format
n_slice, n_coor, _ = coor.shape
r = np.arange(n_coor)
dist = np.zeros([n_slice, n_coor, n_coor])
with concurrent.futures.ThreadPoolExecutor(max_workers=n_cpu) as executor:
for i in xrange(n_slice):
executor.submit(get_dist, cool[i], dist[i], r)
It is possible this problem is not well suited to Dask since there are no inter-chunk computations. 这个问题可能不适合Dask,因为没有块间计算。
map_blocks
The map_blocks method may be helpful: map_blocks方法可能会有所帮助:
dcoor.map_blocks(pdist)
It looks like you're doing a bit of fancy slicing to insert particular values into particular locations of an output array. 看起来你正在做一些花哨的切片,将特定值插入到输出数组的特定位置。 This will probably be awkward to do with dask.arrays. 这可能与dask.arrays有些尴尬。 Instead, I recommend making a function that produces a numpy array 相反,我建议制作一个产生numpy数组的函数
def myfunc(chunk):
values = pdist(chunk[0, :, :])
output = np.zeroes((2048, 2048))
r = np.arange(2048)
output[r[:, None] < r] = values
return output
dcoor.map_blocks(myfunc)
delayed
Worst case scenario you can always use dask.delayed 在最糟糕的情况下,您始终可以使用dask.delayed
from dask import delayed, compute
coor2 = delayed(coor)
slices = [coor2[i] for i in range(coor.shape[0])]
slices2 = [delayed(pdist)(slice) for slice in slices]
results = compute(*slices2)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.