Drop columns in a pandas dataframe based on the % of null values

Question

I have a dataframe with around 60 columns and 2 million rows. Some of the columns are mostly empty. I calculated the % of null values in each column using this function.

def missing_values_table(df): 
    mis_val = df.isnull().sum()
    mis_val_percent = 100 * df.isnull().sum()/len(df)
    mis_val_table = pd.concat([mis_val, mis_val_percent], axis=1)
    mis_val_table_ren_columns = mis_val_table.rename(
    columns = {0 : 'Missing Values', 1 : '% of Total Values'})
    return mis_val_table_ren_columns

Now I want to drop the columns that have more than 80%(for example) values missing. I tried the following code but it does not seem to be working.

df = df.drop(df.columns[df.apply(lambda col: col.isnull().sum()/len(df) > 0.80)], axis=1)

Thank you in advance. Hope I'm not missing something very basic

I receive this error

TypeError: ("'generator' object is not callable", u'occurred at index Unique_Key')

Answer 1

You can use dropna() with threshold parameter

thresh = len(df) * .2
df.dropna(thresh = thresh, axis = 1, inplace = True)

Answer 2

def missing_values(df, percentage):

    columns = df.columns
    percent_missing = df.isnull().sum() * 100 / len(df)
    missing_value_df = pd.DataFrame({'column_name': columns,
                                 'percent_missing': percent_missing})

    missing_drop = list(missing_value_df[missing_value_df.percent_missing>percentage].column_name)
    df = df.drop(missing_drop, axis=1)
    return df