分别为熊猫中的不同列填写缺失值

Question

Say I have a dataframe with different types of columns - numeric and categorical. I want to fill in median values for numeric columns and sample random value for categorical.

This is what I am doing so far:

def fill_nulls(df, num_cols, cat_cols):    
    for col in num_cols:
            dic[col] = 'median'
    for col in cat_cols:
            dic[col] = lambda x: x.sample(1)

    df = df.apply(dic) #NOT SURE WHAT MUST BE HERE

I am creating a dictionary specifying the desired methods for each column. But right now I am not sure how to make this work for missing values. I believe it should be something like apply(dic) ,but I am not sure how to make this to be applied for missing values only.

Thanks!

EDIT:

What I am doing currently:

for col in cat_cols:
    bools = pd.notnull(df[col])
    notnulls = df[col][bools]
    sample = notnulls.sample(1)
    sample = sample.tolist()[0]
    df[col] = df[col].fillna(value=sample)

for col in num_cols:
    med = df[col].median()
    print(type(med))
    df[col] = df[col].fillna(value=med)

It is probably not the most efficient way of doing it. So if anyone knows better way it would be nice to know! thanks!

Answer 1

I have assumed here that your data consists only of numeric and categorical columns (no datetime columns). To demonstrate, first set up some sample data:

import numpy as np
import pandas as pd

df = pd.DataFrame({0: ["0:00", np.nan, "12:00", np.nan, "06:00"],
                  1: [np.nan, 4, 12, 2, np.nan],
                  2: [100, 2, np.nan, -3.6, np.nan],
                  3: ["a", "b", "a", np.nan, np.nan]})
df
    0       1     2     3
0   0:00    NaN   100.0 a
1   NaN     4.0   2.0   b
2   12:00   12.0  NaN   a
3   NaN     2.0   -3.6  NaN
4   06:00   NaN   NaN   NaN

Now, fill in missing values as per your requirements:

# Fill numeric types with median
df = df.fillna(df.median())

# Fill rest of columns (categorical) with random value
df.apply(lambda x: x.fillna(np.random.choice(x[~x.isnull()])))

df
    0       1     2     3
0   0:00    4.0   100.0 a
1   0:00    4.0   2.0   b
2   12:00   12.0  2.0   a
3   0:00    2.0   -3.6  a
4   06:00   4.0   2.0   a