[英]Find and replace outliers with nan in Python
I started to use python and i am trying to find outliers per year using the quantile my data is organized as follows: columns of years, and for each year i have months and their corresponding salinity and temperature我开始使用python,我试图使用分位数每年查找异常值,我的数据组织如下:年份的列,并且每年我都有几个月及其相应的盐度和温度
year=[1997:2021]
month=[1,2...]
SAL=[33,32,50,......,35,...]
Following is my code:以下是我的代码:
#1st quartile
Q1 = DF['SAL'].quantile(0.25)
#3rd quartile
Q3 = DF['SAL'].quantile(0.75)
#calculate IQR
IQR = Q3 - Q1
print(IQR)
df_out = DF['SAL'][((DF['SAL'] < (Q1 - 1.5 * IQR)) |(DF['SAL'] > (Q3 + 1.5 * IQR)))]
I want to identify the month and year of the outlier and replace it with nan.我想识别异常值的月份和年份并将其替换为 nan。
To get the outliers per year , you need to compute the quartiles for each year via groupby
.要获得每年的异常值,您需要通过groupby
计算每年的四分位数。 Other than that, there's not much to change in your code, but I recently learned about between
which seems useful here:除此之外,您的代码没有太大变化,但我最近了解到这between
似乎很有用:
import numpy as np
clean_data = list()
for year, group in DF.groupby('year'):
Q1 = group['SAL'].quantile(0.25)
Q3 = group['SAL'].quantile(0.75)
IQR = Q3 - Q1
# set all values to np.nan that are not (~) in between the two values
group.loc[~group['SAL'].between(Q1 - 1.5 * IQR,
Q3 + 1.5 * IQR,
inclusive=False),
'SAL'] = np.nan
clean_data.append(group)
clean_df = pd.concat(clean_data)
You can use the following function.您可以使用以下功能。 It uses the definition of an outlier that is below Q1-1.5IQR or above Q3+1.5IQR, such as classically done for boxplots.它使用低于 Q1-1.5IQR 或高于 Q3+1.5IQR 的异常值的定义,例如经典的箱线图。
import pandas as pd
import numpy as np
df = pd.DataFrame({'year': np.repeat(range(1997,2022), 12),
'month': np.tile(range(12), 25)+1,
'SAL': np.random.randint(20,40, size=12*25)+np.random.choice([0,-20, 20], size=12*25, p=[0.9,0.05,0.05]),
})
def outliers(s, replace=np.nan):
Q1, Q3 = np.percentile(s, [25 ,75])
IQR = Q3-Q1
return s.where((s > (Q1 - 1.5 * IQR)) & (s < (Q3 + 1.5 * IQR)), replace)
# add new column with excluded outliers
df['SAL_excl'] = df.groupby('year')['SAL'].apply(outliers)
with outliers:有异常值:
import seaborn as sns
sns.boxplot(data=df, x='year', y='SAL')
without outliers:没有异常值:
sns.boxplot(data=df, x='year', y='SAL_excl')
NB.注意。 it is possible that new outliers appear as data has now new Q1/Q3/IQR due to the filtering.由于过滤,数据现在有新的 Q1/Q3/IQR,因此可能会出现新的异常值。
How to retrieve rows with outliers:如何检索具有异常值的行:
df[df['SAL_excl'].isna()]
output:输出:
year month SAL SAL_excl
28 1999 5 53 NaN
33 1999 10 7 NaN
94 2004 11 52 NaN
100 2005 5 38 NaN
163 2010 8 6 NaN
182 2012 3 25 NaN
188 2012 9 22 NaN
278 2020 3 53 NaN
294 2021 7 9 NaN
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.