[英]How to sum certain values in a pandas column DataFrame in a specific date range
[英]Pandas how to filter dataframe with certain range of numbers in a column of dataframe
我正在尝试想出一种方法来过滤 dataframe,以便它只包含进一步处理所需的特定范围的数字。 下面是一个例子 dataframe
data_sample = [['part1', 234], ['part2', 224], ['part3', 214],['part4', 114],['part5', 1111],
['part6',1067],['part7',1034],['part8',1457],['part9', 789],['part10',1367],
['part11',467],['part12',367]
]
data_df = pd.DataFrame(data_sample, columns = ['partname', 'sbin'])
data_df['sbin'] = pd.to_numeric(data_df['sbin'], errors='coerce', downcast='integer')
对于上面的 dataframe,我想过滤掉 sbin 在 [200-230] 和 [1000-1150] 以及 [350-370] 和 [100-130] 范围内的任何部分。
我有一个更大的 dataframe 有更多的范围要删除,因此需要比使用下面的命令更快的方法
data_df.loc[~( ((data_df.sbin >=200) & (data_df.sbin <= 230)) | ((data_df.sbin >=100) & (data_df.sbin <= 130)) | ((data_df.sbin >=350) & (data_df.sbin <= 370))| ((data_df.sbin >=1000) & (data_df.sbin <= 1150)))]
产生 output 如下
partname sbin
0 part1 234
7 part8 1457
8 part9 789
9 part10 1367
10 part11 467
上述方法需要很多条件并且需要很长时间,我想知道是否有更好的方法使用正则表达式或其他一些我不知道的 python 方式。
任何帮助都会很棒
pd.cut
在这里工作正常,特别是当你的间隔不重叠时:
intervals = pd.IntervalIndex.from_tuples([(200, 230), (1000, 1150), (350, 370), (100, 130)])
# if the values do not fall within the intervals, it is a null
# hence the isna check to keep only the null matches
# thanks to @corralien for the include_lowest=True suggestion
data_df.loc[pd.cut(data_df.sbin, intervals, include_lowest=True).isna()]
partname sbin
0 part1 234
7 part8 1457
8 part9 789
9 part10 1367
10 part11 467
新版本
在范围内使用np.logical_and
和any
到 select 的值,并反转掩码以保留其他值。
intervals = [(100, 130), (200, 230), (350, 370), (1000, 1150)]
m = np.any([np.logical_and(data_df['sbin'] >= l, data_df['sbin'] <= u)
for l, u in intervals], axis=0)
out = data_df.loc[~m]
注意any
可以替换为np.logical_or.reduce
:
intervals = [(100, 130), (200, 230), (350, 370), (1000, 1150)]
m = np.logical_or.reduce([np.logical_and(data_df['sbin'] >= l, data_df['sbin'] <= u)
for l, u in intervals])
out = data_df.loc[~m]
Output 结果:
>>> out
partname sbin
0 part1 234
7 part8 1457
8 part9 789
9 part10 1367
10 part11 467
旧版
不能按原样使用浮点数
使用np.where
和in1d
:
intervals = [(100, 130), (200, 230), (350, 370), (1000, 1150)]
m = np.hstack([np.arange(l, u+1) for l, u in intervals])
out = data_df.loc[~np.in1d(data_df['sbin'], m)]
性能:对于 100k 条记录:
data_df = pd.DataFrame({'sbin': np.random.randint(0, 2000, 100000)})
def exclude_range_danimesejo():
intervals = sorted([(200, 230), (1000, 1150), (350, 370), (100, 130)])
intervals = np.array(intervals).flatten()
mask = (np.searchsorted(intervals, data_df['sbin']) % 2 == 0) & ~np.in1d(data_df['sbin'], intervals[::2])
return data_df.loc[mask]
def exclude_range_sammywemmy():
intervals = pd.IntervalIndex.from_tuples([(200, 230), (1000, 1150), (350, 370), (100, 130)])
return data_df.loc[pd.cut(data_df.sbin, intervals, include_lowest=True).isna()]
def exclude_range_corralien():
intervals = [(100, 130), (200, 230), (350, 370), (1000, 1150)]
m = np.hstack([np.arange(l, u+1) for l, u in intervals])
return data_df.loc[~np.in1d(data_df['sbin'], m)]
def exclude_range_corralien2():
intervals = [(100, 130), (200, 230), (350, 370), (1000, 1150)]
m = np.any([np.logical_and(data_df['sbin'] >= l, data_df['sbin'] <= u)
for l, u in intervals], axis=0)
return data_df.loc[~m]
>>> %timeit exclude_range_danimesejo()
2.66 ms ± 18.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
>>> %timeit exclude_range_sammywemmy()
63.6 ms ± 549 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
>>> %timeit exclude_range_corralien()
6.87 ms ± 58.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
>>> %timeit exclude_range_corralien2()
2.26 ms ± 8.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
对于非重叠间隔,您可以使用np.searchsorted
:
# sort the non overlapping intervals
intervals = sorted([(200, 230), (1000, 1150), (350, 370), (100, 130)])
# flatten
intervals = np.array(intervals).flatten()
# search in the intervals, if the index is even is not in the intervals
mask = (np.searchsorted(intervals, data_df['sbin']) % 2 == 0) & ~np.in1d(data_df['sbin'], intervals[::2])
print(data_df.loc[mask])
Output
partname sbin
0 part1 234
7 part8 1457
8 part9 789
9 part10 1367
10 part11 467
时序(在 3.1 GHz Intel Core i7 上)
%timeit exclude_range_danimesejo()
3.45 ms ± 13.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
设置(除了@Corralien 的设置)
def exclude_range_danimesejo():
intervals = sorted([(200, 230), (1000, 1150), (350, 370), (100, 130)])
intervals = np.array(intervals).flatten()
mask = (np.searchsorted(intervals, data_df['sbin']) % 2 == 0) & ~np.in1d(data_df['sbin'], intervals[::2])
return data_df.loc[mask]
这种方法的总体复杂度为O((n + m) log n)
,其中n
是区间列表的大小, m
是sbin
列的长度。
使用循环来应用间隔条件怎么样?
# this will be broadcast to a boolean array where it is `True` if sbin
# is in any of the intervals in the list and `False` otherwise
is_in_intervals = False
intervals = [(200, 230), (1000, 1150), (350, 370), (100, 130)]
for interval in intervals:
is_in_intervals |= (data_df.sbin >= interval[0]) & (data_df.sbin <= interval[1])
data_df[~is_in_intervals]
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.