[英]Drop columns in a pandas dataframe based on the % of null values
I have a dataframe with around 60 columns and 2 million rows. 我有一个大约60列和200万行的数据帧。 Some of the columns are mostly empty.
有些列大多是空的。 I calculated the % of null values in each column using this function.
我使用此函数计算了每列中的空值百分比。
def missing_values_table(df):
mis_val = df.isnull().sum()
mis_val_percent = 100 * df.isnull().sum()/len(df)
mis_val_table = pd.concat([mis_val, mis_val_percent], axis=1)
mis_val_table_ren_columns = mis_val_table.rename(
columns = {0 : 'Missing Values', 1 : '% of Total Values'})
return mis_val_table_ren_columns
Now I want to drop the columns that have more than 80%(for example) values missing. 现在我想删除缺少80%以上(例如)值的列。 I tried the following code but it does not seem to be working.
我尝试了以下代码但它似乎没有工作。
df = df.drop(df.columns[df.apply(lambda col: col.isnull().sum()/len(df) > 0.80)], axis=1)
Thank you in advance. 先感谢您。 Hope I'm not missing something very basic
希望我不会遗漏一些非常基本的东西
I receive this error 我收到此错误
TypeError: ("'generator' object is not callable", u'occurred at index Unique_Key')
TypeError :(“'generator'对象不可调用”,u'Ccurred在索引Unique_Key')
You can use dropna() with threshold parameter 您可以使用带有阈值参数的dropna()
thresh = len(df) * .2
df.dropna(thresh = thresh, axis = 1, inplace = True)
def missing_values(df, percentage):
columns = df.columns
percent_missing = df.isnull().sum() * 100 / len(df)
missing_value_df = pd.DataFrame({'column_name': columns,
'percent_missing': percent_missing})
missing_drop = list(missing_value_df[missing_value_df.percent_missing>percentage].column_name)
df = df.drop(missing_drop, axis=1)
return df
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.