Pandas 如何在 dataframe 的列中用一定范圍的數字過濾 dataframe

Question

我正在嘗試想出一種方法來過濾 dataframe，以便它只包含進一步處理所需的特定范圍的數字。 下面是一個例子 dataframe

data_sample = [['part1', 234], ['part2', 224], ['part3', 214],['part4', 114],['part5', 1111],
                ['part6',1067],['part7',1034],['part8',1457],['part9', 789],['part10',1367],
                ['part11',467],['part12',367]
        ]
data_df = pd.DataFrame(data_sample, columns = ['partname', 'sbin'])
data_df['sbin'] = pd.to_numeric(data_df['sbin'], errors='coerce', downcast='integer')

對於上面的 dataframe，我想過濾掉 sbin 在 [200-230] 和 [1000-1150] 以及 [350-370] 和 [100-130] 范圍內的任何部分。

我有一個更大的 dataframe 有更多的范圍要刪除，因此需要比使用下面的命令更快的方法

data_df.loc[~( ((data_df.sbin >=200) & (data_df.sbin <= 230)) | ((data_df.sbin >=100) & (data_df.sbin <= 130)) | ((data_df.sbin >=350) & (data_df.sbin <= 370))| ((data_df.sbin >=1000) & (data_df.sbin <= 1150)))]

產生 output 如下

    partname    sbin
0   part1       234
7   part8       1457
8   part9       789
9   part10      1367
10  part11      467

上述方法需要很多條件並且需要很長時間，我想知道是否有更好的方法使用正則表達式或其他一些我不知道的 python 方式。

任何幫助都會很棒

Answer 1

pd.cut在這里工作正常，特別是當你的間隔不重疊時：

intervals = pd.IntervalIndex.from_tuples([(200, 230), (1000, 1150), (350, 370), (100, 130)])

# if the values do not fall within the intervals, it is a null
# hence the isna check to keep only the null matches
# thanks to @corralien for the include_lowest=True suggestion

data_df.loc[pd.cut(data_df.sbin, intervals, include_lowest=True).isna()]

   partname  sbin
0     part1   234
7     part8  1457
8     part9   789
9    part10  1367
10   part11   467

Answer 2

新版本

在范圍內使用np.logical_and和any到 select 的值，並反轉掩碼以保留其他值。

intervals = [(100, 130), (200, 230), (350, 370), (1000, 1150)]
m = np.any([np.logical_and(data_df['sbin'] >= l, data_df['sbin'] <= u)
                                    for l, u in intervals], axis=0)
out = data_df.loc[~m]

注意any可以替換為np.logical_or.reduce ：

intervals = [(100, 130), (200, 230), (350, 370), (1000, 1150)]
m = np.logical_or.reduce([np.logical_and(data_df['sbin'] >= l, data_df['sbin'] <= u)
                                    for l, u in intervals])
out = data_df.loc[~m]

Output 結果：

>>> out
   partname  sbin
0     part1   234
7     part8  1457
8     part9   789
9    part10  1367
10   part11   467

---

舊版

不能按原樣使用浮點數

使用np.where和in1d ：

intervals = [(100, 130), (200, 230), (350, 370), (1000, 1150)]
m = np.hstack([np.arange(l, u+1) for l, u in intervals])
out = data_df.loc[~np.in1d(data_df['sbin'], m)]

---

性能：對於 100k 條記錄：

data_df = pd.DataFrame({'sbin': np.random.randint(0, 2000, 100000)})

def exclude_range_danimesejo():
    intervals = sorted([(200, 230), (1000, 1150), (350, 370), (100, 130)])
    intervals = np.array(intervals).flatten()
    mask = (np.searchsorted(intervals, data_df['sbin']) % 2 == 0) & ~np.in1d(data_df['sbin'], intervals[::2])
    return data_df.loc[mask]

def exclude_range_sammywemmy():
    intervals = pd.IntervalIndex.from_tuples([(200, 230), (1000, 1150), (350, 370), (100, 130)])
    return data_df.loc[pd.cut(data_df.sbin, intervals, include_lowest=True).isna()]

def exclude_range_corralien():
    intervals = [(100, 130), (200, 230), (350, 370), (1000, 1150)]
    m = np.hstack([np.arange(l, u+1) for l, u in intervals])
    return data_df.loc[~np.in1d(data_df['sbin'], m)]

def exclude_range_corralien2():
    intervals = [(100, 130), (200, 230), (350, 370), (1000, 1150)]
    m = np.any([np.logical_and(data_df['sbin'] >= l, data_df['sbin'] <= u)
                                        for l, u in intervals], axis=0)
    return data_df.loc[~m]

>>> %timeit exclude_range_danimesejo()
2.66 ms ± 18.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

>>> %timeit exclude_range_sammywemmy()
63.6 ms ± 549 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

>>> %timeit exclude_range_corralien()
6.87 ms ± 58.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

>>> %timeit exclude_range_corralien2()
2.26 ms ± 8.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Answer 3

對於非重疊間隔，您可以使用np.searchsorted ：

# sort the non overlapping intervals
intervals = sorted([(200, 230), (1000, 1150), (350, 370), (100, 130)])

# flatten
intervals = np.array(intervals).flatten()

# search in the intervals, if the index is even is not in the intervals
mask = (np.searchsorted(intervals, data_df['sbin']) % 2 == 0) & ~np.in1d(data_df['sbin'], intervals[::2])

print(data_df.loc[mask])

Output

   partname  sbin
0     part1   234
7     part8  1457
8     part9   789
9    part10  1367
10   part11   467

時序（在 3.1 GHz Intel Core i7 上）

%timeit exclude_range_danimesejo()
3.45 ms ± 13.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

設置（除了@Corralien 的設置）

def exclude_range_danimesejo():
    intervals = sorted([(200, 230), (1000, 1150), (350, 370), (100, 130)])
    intervals = np.array(intervals).flatten()
    mask = (np.searchsorted(intervals, data_df['sbin']) % 2 == 0) & ~np.in1d(data_df['sbin'], intervals[::2])
    return data_df.loc[mask]

這種方法的總體復雜度為O((n + m) log n) ，其中n是區間列表的大小， m是sbin列的長度。

Answer 4

使用循環來應用間隔條件怎么樣？

# this will be broadcast to a boolean array where it is `True` if sbin
# is in any of the intervals in the list and `False` otherwise
is_in_intervals = False
intervals = [(200, 230), (1000, 1150), (350, 370), (100, 130)]

for interval in intervals:
    is_in_intervals |= (data_df.sbin >= interval[0]) & (data_df.sbin <= interval[1])

data_df[~is_in_intervals]