[英]How to efficiently find separately for each element N maximum values among multiple matrices?
I am looping through a large number of H x W matrices.我正在遍历大量 H x W 矩阵。 I cannot store them all in memory.
我无法将它们全部存储在 memory 中。 I need to get N matrices.
我需要得到 N 个矩阵。 For example, the element of the 1st of N matrix in position (i, j) will be the largest among all elements in position (i, j) of all processed matrix matrices.
例如,position (i, j) 中 N 矩阵的第一个元素将是所有已处理矩阵矩阵中 position (i, j) 中所有元素中最大的元素。 For the second of the N matrix, the elements that are the second-largest will be taken, and so on.
对于 N 矩阵中的第二个,将取第二大的元素,依此类推。
Example.例子。
Let N = 2. Then the 1st matrix will look like this.让 N = 2。那么第一个矩阵将如下所示。
And the second matrix is like this.第二个矩阵是这样的。
How to do such an operation inside a loop so as not to store all matrices in memory?如何在循环内进行这样的操作,以免将所有矩阵存储在 memory 中?
The comments suggested using the np.partition
function.评论建议使用
np.partition
function。 I replaced the use of numpy with cupy , which uses the GPU.我用 cupy 替换了numpy的使用,它使用了 GPU。 And also added a buffer to sort less frequently.
并且还添加了一个缓冲区来减少排序频率。
import cupy as np
buf = // # As much as fits into the GPU
largests = np.zeros((buf + N, h, w))
for i in range(num):
val = //
largests[i % buf] = val
if i % buf == buf - 1:
largests.partition(range(buf, buf + N), axis=0)
largests.partition(range(buf, buf + N), axis=0) # Let's not forget the tail
res = largests[:-(N + 1):-1]
The solution does not work very quickly, but I have come to terms with this speed.该解决方案的工作速度不是很快,但我已经接受了这种速度。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.