检测多个numpy 2-d数组中的第一个唯一行

Question

我有多个numpy 2-d数组，我想比较行。 我的函数的输出应该是一个numpy二维数组，表示三个输入数组的所有行。 我希望能够检测到第一次出现行，每秒或第三个重复行应该在输出中标记为False。 单个数组中不可能有重复的行。

如果有可能我想避免使用循环，因为它们会降低计算速度。

例：

array1 = array([[444, 427],
   [444, 428],
   [444, 429],
   [444, 430],
   [445, 421]], dtype=uint64)

array2 = array([[446, 427],
   [446, 440],
   [444, 429],
   [444, 432],
   [445, 421]], dtype=uint64)

array3 = array([[447, 427],
   [446, 441],
   [444, 429],
   [444, 432],
   [445, 421]], dtype=uint64)

# output
array([[True, True, True, True,  True],
   [ True,  True,  False, True,  False],
   [ True,  True,  False, False,  False]], dtype=bool)

有任何想法吗？

Answer 1

这是一种快速矢量化方法：

def find_dupe_rows(*arrays):

    A = np.vstack(arrays)
    rtype = np.dtype((np.void, A.dtype.itemsize*A.shape[1]))
    _, first_idx = np.unique(A.view(rtype), return_index=True)
    out = np.zeros(A.shape[0], np.bool)
    out[first_idx] = True

    return out.reshape(len(arrays), -1)

用法示例：

print(find_dupe_rows(array1, array2, array3))
# [[ True  True  True  True  True]
#  [ True  True False  True False]
#  [ True  True False False False]]

---

为了打破这一点：

1. 堆叠三个子阵列以生成(15, 2) 15,2 (15, 2)数组：

    A = np.vstack((array1, array2, array3)) 

2. 使用np.unique和这个技巧有效地找到每个唯一行首先出现在A的索引：

    rtype = np.dtype((np.void, A.dtype.itemsize * A.shape[1])) _, first_idx = np.unique(A.view(rtype), return_index=True) 

3. 不是第一次出现的唯一行的每一行都可以视为重复：

    out = np.zeros(A.shape[0], np.bool) # output is False by default out[first_idx] = True # set first occurrences to True 

4. 最后，根据您的示例输出，将此布尔矢量重新(narrays, nrows)为(narrays, nrows) ：

    return out.reshape(len(arrays), -1) 

Answer 2

如果您正在寻找2D输入数组中相同行ID的重复项，这里是一个矢量化方法 -

# Concatenate all input 2D arrays to form a single tall 2D array
A = np.vstack((array1,array2,array3))

# Consider the rows of the 2D input arrays as linear indexing tuples. 
# Thus, we can reduce the input size, reduced by length of rows.
# This would help in simplifying the solution ahead and help in performance.
A_lidx = np.ravel_multi_index(A.T,A.max(0)+1).reshape(-1,array1.shape[0])

# Finally use broadcasting to perform elementwise comparison between 
# the elements of each column against themselves. Then, use argmax 
# along the first axis giving us the first indices of the duplicates, 
# which when compared with index ID would lead us to final boolean array.
out = (A_lidx[:,None] == A_lidx).argmax(0) == np.arange(A_lidx.shape[0])[:,None]

如果您正在寻找跨所有2D输入数组的所有行的全局搜索，您需要一些修改过的方法，如下所示 -

# Concatenate all input 2D arrays to form a single tall 2D array
A = np.vstack((array1,array2,array3))

# Consider the rows of the 2D input arrays as linear indexing tuples. 
# Thus, we can reduce the input size, reduced by length of rows.
# This would help in simplifying the solution ahead and help in performance.
A_lidx = np.ravel_multi_index(A.T,A.max(0)+1).ravel()

# Find first occurances by differentiating sorted version and 
# looking for indices with positive change.
sidx = A_lidx.argsort()
first_occ = sidx[np.append(0,np.where(np.diff(A_lidx[sidx])>0)[0]+1)]

# Finally, set those indices as True in an output array of appropriate length
out = np.in1d(np.arange(len(A_lidx)),first_occ).reshape(3,-1)

请注意，计算first_occ的步骤基本上是使用`@ ali_m解决方案中使用的np.unique(..., return_index=True) 粗略方法 。

Answer 3

array1 = np.array([[444, 427],
   [444, 428],
   [444, 429],
   [444, 430],
   [445, 421]])

array2 = np.array([[446, 427],
   [446, 440],
   [444, 429],
   [444, 432],
   [445, 421]])

array3 = np.array([[447, 427],
   [446, 441],
   [444, 429],
   [444, 432],
   [445, 421]])

array_list = [array1,array2,array3]
x = set()
check = np.ones((len(array_list),array1.shape[0]),dtype=bool)
for i,item in enumerate(array_list):
    for j in range(item.shape[0]):
        temp = tuple(item[j,:])
        if temp in x:
            check[i,j] = False
        else:
            x.add(temp)

print check

希望这会有所帮助。