[英]Numpy: how to extract rows of numpy array between multiple pairs of values?
I am relatively new to Python and I am currently facing some problems in implementing a conceptually simple algorithm in an efficient way.我对 Python 比较陌生,目前在以有效的方式实现概念上简单的算法时遇到了一些问题。 I have been able to do it in pandas (but it is quite slow to execute).我已经能够在 pandas 中做到这一点(但执行起来很慢)。
I have a ndarray composed by n rows and 3 columns:我有一个由 n 行和 3 列组成的 ndarray:
--------------------
"A" | 1 | 12
--------------------
"B" | 2 | 34
--------------------
"S" | 3 | 1
--------------------
"B" | 4 | 145
--------------------
"A" | 5 | 132
--------------------
"B" | 6 | 234
--------------------
"E" | 7 | 1
--------------------
"B" | 8 | 15
--------------------
The first column represents an id , the second column a timestamp and the third a value .第一列表示一个id ,第二列表示时间戳,第三列表示值。 I have to filter the ndarray taking only the rows with timestamps included between the id "S" (Start) timestamp and the id "E" (End) timestamp.我必须过滤 ndarray,只取包含在 id“S”(开始)时间戳和 id“E”(结束)时间戳之间的时间戳的行。 It is possible o have more than one pair of "S" and "E" in the same ndarray.在同一个 ndarray 中可能有不止一对“S”和“E”。 In case of non-consecutive "S" and "E" pair, I need the shortest subarray.在不连续的“S”和“E”对的情况下,我需要最短的子数组。 In other words, no id "S" or "E" should appear in output.换句话说,在 output 中不应出现 id "S" 或 "E"。
So the output should be:所以 output 应该是:
--------------------
"B" | 4 | 145
--------------------
"A" | 5 | 132
--------------------
"B" | 6 | 234
--------------------
As already said, I have obtained this result using pandas, but the function is really long and complicated and it executes really slowly.如前所述,我使用 pandas 获得了这个结果,但是 function 真的很长很复杂,而且执行起来很慢。 So I am sure that with numpy it is possible to obtain a better and most efficient algorithm.所以我确信使用 numpy 可以获得更好和最有效的算法。
Do you have any ideas?你有什么想法?
Thanks in advance.提前致谢。
EDIT编辑
Here it is a code snippet which obtains expected results using pandas.这是一个使用 pandas 获得预期结果的代码片段。 The execution time is about 0.015 seconds on a "Intel Core i7-6820 @ 2.70GHz" processor.在“Intel Core i7-6820 @ 2.70GHz”处理器上执行时间约为 0.015 秒。
df = pd.DataFrame({'id': ['A', 'B', 'C', 'S', 'C', 'C', 'A', 'B', 'E', 'A', 'C', 'B', 'B', 'S', 'C', 'A', 'E', 'B'],
't': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18],
'v': [145, 543, 12, 1, 14, 553, 65, 657, 1, 32, 54, 22, 11, 1, 6, 22, 1, 4]})
print(df)
res = pd.DataFrame()
id = "id"
t = "t"
v = "v"
id_bit_start = "S"
id_bit_end = "E"
# taking only "S" and "E" from df (when their value is 1)
df_data_bit = df.loc[
((df[id] == id_bit_start) |
(df[id] == id_bit_end)) &
(df[v] == 1.0)
]
# do something only if at least one "S" is present
if id_bit_start in df_data_bit[id].values:
# creating empty list of time windows
t_windows = list()
t_wind_temp = [None, None]
# for each bit "S" or "E"
for index, bit in df_data_bit.iterrows():
# if it is a "S"
if bit[id] == id_bit_start:
# set the start of the time window
t_wind_temp[0] = bit[t]
# if it is a "E" and the "S" has already been processed
elif t_wind_temp[0] is not None:
# set the end of the time window
t_wind_temp[1] = bit[t]
# append the current time window to our list
t_windows.append(t_wind_temp)
# reset the current time window
t_wind_temp = [None, None]
# taking everything but "S" and "E"
df_data = df.loc[
~((df[id] == id_bit_start) |
(df[id] == id_bit_end))
]
# for each created time window
for t_window in t_windows:
# take only data with timestamps between the time window
result = df_data.loc[
(df_data[t] >= t_window[0])
&
(df_data[t] <= t_window[1])
]
# append to the final result
res = pd.concat([res, result])
print(res)
This takes care of your uncertainty of consecutiveness of S and E:这可以解决您对 S 和 E 连续性的不确定性:
Assuming your timestamps are in ascending order:假设您的时间戳按升序排列:
import re
a = df.to_records(index=False)
idx = [m.span() for m in re.finditer('S[^{SE}]*?E', ''.join(a['id']))]
indexer = np.r_[tuple([np.s_[i+1:j-1] for (i,j) in idx])]
a_filtered = a[indexer]
Explanation :说明:
There are some tricks in fast calculating this:快速计算有一些技巧:
S.*?E
(note you can change S and E to any substrings if your id is not single letter)寻找S.*?E
的非贪婪匹配(请注意,如果您的 id 不是单个字母,您可以将 S 和 E 更改为任何子字符串)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.