[英]numpy 1D array: mask elements that repeat more than n times
Q: given an array of integers like问:给定一个整数数组,如
[1, 1, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5]
I need to mask elements that repeat more than N
times.我需要屏蔽重复超过
N
次的元素。 The goal is to retrieve the boolean mask array.目标是检索 boolean 掩码数组。
I came up with a rather complicated solution:我想出了一个相当复杂的解决方案:
import numpy as np
bins = np.array([1, 1, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5])
N = 3
splits = np.split(bins, np.where(np.diff(bins) != 0)[0]+1)
mask = []
for s in splits:
if s.shape[0] <= N:
mask.append(np.ones(s.shape[0]).astype(np.bool_))
else:
mask.append(np.append(np.ones(N), np.zeros(s.shape[0]-N)).astype(np.bool_))
mask = np.concatenate(mask)
giving eg给例如
bins[mask]
Out[90]: array([1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 5, 5, 5])
Is there a nicer way to do this?有没有更好的方法来做到这一点?
Wrap-up: Here's a slim version of MSeifert's benchmark plot (thanks for pointing me to simple_benchmark
).总结:这是 MSeifert 基准 plot 的精简版(感谢您指向
simple_benchmark
)。 Showing the four most performant options:显示四个性能最高的选项:
The idea proposed by Florian H , modified by Paul Panzer seems to be a great way of solving this problem as it is pretty straight forward and numpy
-only. Florian H提出的想法,由Paul Panzer修改,似乎是解决这个问题的好方法,因为它非常简单,并且
numpy
。 If you're fine with using numba
, MSeifert's solution outperforms the other.如果您对使用
numba
感到满意,那么 MSeifert 的解决方案会优于其他解决方案。
I chose to accept MSeifert's answer as solution as it is the more general answer: It correctly handles arbitrary arrays with (non-unique) blocks of consecutive repeating elements.我选择接受 MSeifert 的答案作为解决方案,因为它是更一般的答案:它正确处理具有(非唯一)连续重复元素块的任意 arrays。 In case
numba
is a no-go, Divakar's answer is also worth a look.如果
numba
, Divakar 的回答也值得一看。
Disclaimer: this is just a sounder implementation of @FlorianH's idea:免责声明:这只是@FlorianH 想法的更合理的实现:
def f(a,N):
mask = np.empty(a.size,bool)
mask[:N] = True
np.not_equal(a[N:],a[:-N],out=mask[N:])
return mask
For larger arrays this makes a huge difference:对于更大的 arrays 这会产生巨大的差异:
a = np.arange(1000).repeat(np.random.randint(0,10,1000))
N = 3
print(timeit(lambda:f(a,N),number=1000)*1000,"us")
# 5.443050000394578 us
# compare to
print(timeit(lambda:[True for _ in range(N)] + list(bins[:-N] != bins[N:]),number=1000)*1000,"us")
# 76.18969900067896 us
Approach #1: Here's a vectorized way -方法#1:这是一种矢量化的方式 -
from scipy.ndimage.morphology import binary_dilation
def keep_N_per_group(a, N):
k = np.ones(N,dtype=bool)
m = np.r_[True,a[:-1]!=a[1:]]
return a[binary_dilation(m,k,origin=-(N//2))]
Sample run -样品运行 -
In [42]: a
Out[42]: array([1, 1, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5])
In [43]: keep_N_per_group(a, N=3)
Out[43]: array([1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 5, 5, 5])
Approach #2: A bit more compact version -方法#2:更紧凑的版本 -
def keep_N_per_group_v2(a, N):
k = np.ones(N,dtype=bool)
return a[binary_dilation(np.ediff1d(a,to_begin=a[0])!=0,k,origin=-(N//2))]
Approach #3: Using the grouped-counts and np.repeat
(won't give us the mask though) -方法#3:使用分组计数和
np.repeat
(虽然不会给我们掩码) -
def keep_N_per_group_v3(a, N):
m = np.r_[True,a[:-1]!=a[1:],True]
idx = np.flatnonzero(m)
c = np.diff(idx)
return np.repeat(a[idx[:-1]],np.minimum(c,N))
Approach #4: With a view-based
method -方法#4:使用基于
view-based
方法 -
from skimage.util import view_as_windows
def keep_N_per_group_v4(a, N):
m = np.r_[True,a[:-1]!=a[1:]]
w = view_as_windows(m,N)
idx = np.flatnonzero(m)
v = idx<len(w)
w[idx[v]] = 1
if v.all()==0:
m[idx[v.argmin()]:] = 1
return a[m]
Approach #5: With a view-based
method without indices from flatnonzero
-方法 #5:使用基于
view-based
方法,没有来自flatnonzero
的索引 -
def keep_N_per_group_v5(a, N):
m = np.r_[True,a[:-1]!=a[1:]]
w = view_as_windows(m,N)
last_idx = len(a)-m[::-1].argmax()-1
w[m[:-N+1]] = 1
m[last_idx:last_idx+N] = 1
return a[m]
I want to present a solution using numba which should be fairly easy to understand.我想提出一个使用numba的解决方案,它应该相当容易理解。 I assume that you want to "mask" consecutive repeating items:
我假设您想“屏蔽”连续重复的项目:
import numpy as np
import numba as nb
@nb.njit
def mask_more_n(arr, n):
mask = np.ones(arr.shape, np.bool_)
current = arr[0]
count = 0
for idx, item in enumerate(arr):
if item == current:
count += 1
else:
current = item
count = 1
mask[idx] = count <= n
return mask
For example:例如:
>>> bins = np.array([1, 1, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5])
>>> bins[mask_more_n(bins, 3)]
array([1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 5, 5, 5])
>>> bins[mask_more_n(bins, 2)]
array([1, 1, 2, 2, 3, 3, 4, 4, 5, 5])
Using simple_benchmark
- however I haven't included all approaches.使用
simple_benchmark
- 但是我没有包括所有方法。 It's a log-log scale:这是一个对数刻度:
It seems like the numba solution cannot beat the solution from Paul Panzer which seems to be faster for large arrays by a bit (and doesn't require an additional dependency).似乎 numba 解决方案无法击败 Paul Panzer 的解决方案,这对于大型 arrays 来说似乎更快(并且不需要额外的依赖)。
However both seem to outperform the other solutions, but they do return a mask instead of the "filtered" array.然而,两者似乎都优于其他解决方案,但它们确实返回一个掩码而不是“过滤”数组。
import numpy as np
import numba as nb
from simple_benchmark import BenchmarkBuilder, MultiArgument
b = BenchmarkBuilder()
bins = np.array([1, 1, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5])
@nb.njit
def mask_more_n(arr, n):
mask = np.ones(arr.shape, np.bool_)
current = arr[0]
count = 0
for idx, item in enumerate(arr):
if item == current:
count += 1
else:
current = item
count = 1
mask[idx] = count <= n
return mask
@b.add_function(warmups=True)
def MSeifert(arr, n):
return mask_more_n(arr, n)
from scipy.ndimage.morphology import binary_dilation
@b.add_function()
def Divakar_1(a, N):
k = np.ones(N,dtype=bool)
m = np.r_[True,a[:-1]!=a[1:]]
return a[binary_dilation(m,k,origin=-(N//2))]
@b.add_function()
def Divakar_2(a, N):
k = np.ones(N,dtype=bool)
return a[binary_dilation(np.ediff1d(a,to_begin=a[0])!=0,k,origin=-(N//2))]
@b.add_function()
def Divakar_3(a, N):
m = np.r_[True,a[:-1]!=a[1:],True]
idx = np.flatnonzero(m)
c = np.diff(idx)
return np.repeat(a[idx[:-1]],np.minimum(c,N))
from skimage.util import view_as_windows
@b.add_function()
def Divakar_4(a, N):
m = np.r_[True,a[:-1]!=a[1:]]
w = view_as_windows(m,N)
idx = np.flatnonzero(m)
v = idx<len(w)
w[idx[v]] = 1
if v.all()==0:
m[idx[v.argmin()]:] = 1
return a[m]
@b.add_function()
def Divakar_5(a, N):
m = np.r_[True,a[:-1]!=a[1:]]
w = view_as_windows(m,N)
last_idx = len(a)-m[::-1].argmax()-1
w[m[:-N+1]] = 1
m[last_idx:last_idx+N] = 1
return a[m]
@b.add_function()
def PaulPanzer(a,N):
mask = np.empty(a.size,bool)
mask[:N] = True
np.not_equal(a[N:],a[:-N],out=mask[N:])
return mask
import random
@b.add_arguments('array size')
def argument_provider():
for exp in range(2, 20):
size = 2**exp
yield size, MultiArgument([np.array([random.randint(0, 5) for _ in range(size)]), 3])
r = b.run()
import matplotlib.pyplot as plt
plt.figure(figsize=[10, 8])
r.plot()
You could do this with indexing.您可以通过索引来做到这一点。 For any N the code would be:
对于任何 N,代码将是:
N = 3
bins = np.array([1, 1, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5,6])
mask = [True for _ in range(N)] + list(bins[:-N] != bins[N:])
bins[mask]
output: output:
array([1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 5, 5, 5, 6]
You could use a while loop that checks if the array element N positions back is equal to the current one.您可以使用 while 循环来检查数组元素 N 位置是否等于当前位置。 Note this solution assumes the array is ordered.
请注意,此解决方案假定数组是有序的。
import numpy as np
bins = [1, 1, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5]
N = 3
counter = N
while counter < len(bins):
drop_condition = (bins[counter] == bins[counter - N])
if drop_condition:
bins = np.delete(bins, counter)
else:
# move on to next element
counter += 1
A much nicer way would be to use numpy
's unique()
-function.更好的方法是使用
numpy
的unique()
-函数。 You will get unique entries in your array and also the count of how often they appear:您将在数组中获得唯一条目以及它们出现的频率:
bins = np.array([1, 1, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5])
N = 3
unique, index,count = np.unique(bins, return_index=True, return_counts=True)
mask = np.full(bins.shape, True, dtype=bool)
for i,c in zip(index,count):
if c>N:
mask[i+N:i+c] = False
bins[mask]
output: output:
array([1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 5, 5, 5])
You could use numpy.unique
.您可以使用
numpy.unique
。 The variable final_mask
can be used to extract the traget elements from the array bins
.变量
final_mask
可用于从数组bins
中提取 traget 元素。
import numpy as np
bins = np.array([1, 1, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5])
repeat_max = 3
unique, counts = np.unique(bins, return_counts=True)
mod_counts = np.array([x if x<=repeat_max else repeat_max for x in counts])
mask = np.arange(bins.size)
#final_values = np.hstack([bins[bins==value][:count] for value, count in zip(unique, mod_counts)])
final_mask = np.hstack([mask[bins==value][:count] for value, count in zip(unique, mod_counts)])
bins[final_mask]
Output : Output :
array([1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 5, 5, 5])
You could use grouby to group common elements and filter list that are longer than N .您可以使用grouby对长于N的常见元素和过滤器列表进行分组。
import numpy as np
from itertools import groupby, chain
def ifElse(condition, exec1, exec2):
if condition : return exec1
else : return exec2
def solve(bins, N = None):
xss = groupby(bins)
xss = map(lambda xs : list(xs[1]), xss)
xss = map(lambda xs : ifElse(len(xs) > N, xs[:N], xs), xss)
xs = chain.from_iterable(xss)
return list(xs)
bins = np.array([1, 1, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5])
solve(bins, N = 3)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.