如何有条件地从 Pandas dataframe 中删除行

Question

我的意图是将主 dataframe purchases拆分为 2 个数据帧：一个正常的数据帧，一个包含取决于 NaN 的异常值的数据帧。 下面的代码应该跨越 dataframe 长度，但它实际上抛出异常IndexError: index 4 is out of bounds for axis 0 with size 3

打印语句表明条件是正确的，但结果（使用for i in range(0,m-1):时）是错误的，这可能是由于删除行的方式：

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
data = {
    'apples': [3, 2, 0, np.nan, 2],
    'oranges': [0, 7, 7, 2, 7],
    'figs':[1, np.nan, 10, np.nan, 10]
}
purchases = pd.DataFrame(data)
purchases = pd.DataFrame(data, index=['June', 'Robert', 'Lily', 'David', 'Bob'])
# calculate the proportion of NaN per row
l = len(purchases.columns)
m = len(purchases)
n_nan=0
W= [0 for w in range(m)]
X = [i for i in range(0,m)]
for i in range(0,m):
    n_nan = purchases.iloc[i,:].isna().sum()
    print('row ',i,' number of NaN ',n_nan,' % of Nan ',n_nan*100/l)
    W[i]=n_nan*100/l
# Write code to divide the data into two subsets based on the number of missing
# values in each row.
# use https://www.shanelynn.ie/using-pandas-dataframe-creating-editing-viewing-data-in-python/
purchases_normal = purchases.copy()
purchases_outliers = purchases.copy()
print('purchases ')
print(purchases)
print('----------------------')
#
for j in range(0,m-0):
#    print('row ',j,' W = ',W[j])
    if W[j]> 20:
        print('at iteration ',j, ' going to drop from purchases_normal as W= ', W[j],' is > 20')
        purchases_normal.drop(purchases_normal.index[j], inplace=True)
    else:
        print('at iteration ',j, ' going to drop from purchases_outliers as W= ', W[j],' is < 20')
        purchases_outliers.drop(purchases_outliers.index[j], inplace=True)
print('purchases normal')
print(purchases_normal)
print('------')
print('purchases outliers')
print(purchases_outliers)

Answer 1

试试下面的循环：

for j in range(0,m-0):
#    print('row ',j,' W = ',W[j])
    if W[j]> 20:
        print('at iteration ',j, ' going to drop from purchases_normal as W= ', W[j],' is > 20')
        purchases_normal = purchases_normal.drop(purchases.index[j])
    else:
        print('at iteration ',j, ' going to drop from purchases_outliers as W= ', W[j],' is < 20')
        purchases_outliers = purchases_outliers.drop(purchases.index[j])

Answer 2

Pandas 的构建方式不必使用for-loop 。 如果你发现自己在使用 for 循环，那么你 98% 的可能性是你做错了。

如果我了解您的目标：
1.逐行查找NaN的数量
2.获取用于逻辑的百分比（X时下降）

# ... 
df['number_nan'] = df.isna().sum(axis=1)
df['pct_nan'] = df['number_na']/len(df.columns)

现在有了这些额外的列。 你可以筛选

above_20 = .2
# dt = df with rows with above 20 percent missing values 
dt = df[df['pct_na'] > above_20]

如果我了解您的目标，请告诉我。