[英]Efficiently counting runs of non-zero values
我正在处理降雨量的时间序列,我想要计算单个降雨事件的长度和体积,其中“事件”是一系列非零时间步长。 我正在处理大约60k时间步的多个时间序列,我目前的方法很慢。
目前我有以下内容:
import numpy as np
def count_events(timeseries):
start = 0
end = 0
lengths = []
volumes = []
# pad a 0 at the edges so as to include edges as "events"
for i, val in enumerate(np.pad(timeseries, pad_width = 1, mode = 'constant')):
if val > 0 and start==0:
start = i
if val == 0 and start>0:
end = i
if end - start != 1:
volumes.append(np.sum(timeseries[start:end]))
elif end - start == 1:
volumes.append(timeseries[start-1])
lengths.append(end-start)
start = 0
return np.asarray(lengths), np.asarray(volumes)
预期产量:
testrain = np.array([1,0,1,0,2,2,8,2,0,0,0.1,0,0,1])
lengths, volumes = count_events(testrain)
print lengths
[1 1 4 1 1]
print volumes
[ 1. 1. 12. 0.1 1. ] # 12 should actually be 14, my code returns wrong results.
我想有一个更好的方法来做到这一点,利用numpy的效率,但没有想到......
编辑:
比较不同的解决方案:
testrain = np.random.normal(10,5, 60000)
testrain[testrain<0] = 0
我的解决方案(产生错误的结果,不完全确定原因):
%timeit count_events(testrain)
#10 loops, best of 3: 129 ms per loop
@耶的:
%timeit dawg(testrain) # using itertools
#10 loops, best of 3: 113 ms per loop
%timeit dawg2(testrain) # using pure numpy
#10 loops, best of 3: 156 ms per loop
@ DSM的:
%timeit DSM(testrain)
#10 loops, best of 3: 28.4 ms per loop
@ DanielLenz的:
%timeit DanielLenz(testrain)
#10 loops, best of 3: 316 ms per loop
这是一个groupby解决方案:
import numpy as np
from itertools import groupby
testrain = np.array([1,0,1,0,2,2,8,2,0,0,0.1,0,0,1])
lengths=[]
volumes=[]
for k, l in groupby(testrain, key=lambda v: v>0):
if k:
li=list(l)
lengths.append(len(li))
volumes.append(sum(li))
print lengths
print volumes
打印
[1, 1, 4, 1, 1]
[1.0, 1.0, 14.0, 0.10000000000000001, 1.0]
如果你想要一些纯粹的numpy:
def find_runs(arr):
subs=np.split(testrain, np.where(testrain== 0.)[0])
arrs=[np.delete(sub, np.where(sub==0.)) for sub in subs]
return [(len(e), sum(e)) for e in arrs if len(e)]
>>> find_runs(testrain)
[(1, 1.0), (1, 1.0), (4, 14.0), (1, 0.10000000000000001), (1, 1.0)]
>>> length, volume=zip(*find_runs(testrain))
虽然你可以在纯粹的numpy中做到这一点,但你基本上将numpy应用于pandas
问题。 你的volume
是groupby操作的结果,你可以伪装成numpy,但它是大熊猫的原生。
例如:
>>> tr = pd.Series(testrain)
>>> nonzero = (tr != 0)
>>> group_ids = (nonzero & (nonzero != nonzero.shift())).cumsum()
>>> events = tr[nonzero].groupby(group_ids).agg([sum, len])
>>> events
sum len
1 1.0 1
2 1.0 1
3 14.0 4
4 0.1 1
5 1.0 1
这是我的方法,使用scipy.ndimage.measurements
labels
:
import numpy as np
from scipy.ndimage.measurements import label
testrain = np.array([1,0,1,0,2,2,8,2,0,0,0.1,0,0,1])
labels, nlabels = label(testrain)
labels
>> array([1, 0, 2, 0, 3, 3, 3, 3, 0, 0, 4, 0, 0, 5], dtype=int32)
def sum_and_length(n):
obj = np.array(testrain[labels==n])
return [np.sum(obj), obj.size]
sums, lengths = np.array(map(sum_and_length, range(1, nlabels+1))).T
sums
>> array([ 1. , 1. , 14. , 0.1, 1. ])
lenghts
>> array([ 1., 1., 4., 1., 1.])
这不是最美丽的方法,因为这个问题对于pandas
来说是完美的,但它可能会让你看看measurements
,这是一个非常强大的工具集。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.