Python pandas：基于图例数据框标记分类值

Question

I have a big dataset (2m rows, 70 variables), which has many categorical variables. All categorical variables are coded in numbers (eg see df1)

df1:
   obs  gender  job
    1     1       1
    2     1       2
    3     2       2
    4     1       1

I have a another data frame with all explanations, looking like this:

df2:
Var:     Value:   Label:
gender     1      male
gender     2      female
job        1      blue collar
job        2      white collar

Is there a fast way to replace all values of the categorical columns with their label from df2? This would save me the work to always look up the meaning of the value in df2. I found some solutions to replace values by hand, but I look for an automatic way doing this.

Thank you

Answer 1

You could use a dictionary generated from df2 . Like this:

Firstly, generating some dummy data:

import pandas as pd
import numpy as np

df1 = pd.DataFrame()
df1['obs'] = range(1,1001)
df1['gender'] = np.random.choice([1,2],1000)
df1['job'] = np.random.choice([1,2],1000)

df2 = pd.DataFrame()
df2['var'] = ['gender','gender','job','job']
df2['value'] = [1,2,1,2]
df2['label'] = ['male','female','blue collar', 'white collar']

If you want to replace one variable something like this:

genderDict = dict(df2.loc[df2['var']=='gender'][['value','label']].values)
df1['gender_name'] = df1['gender'].apply(lambda x: genderDict[x])

And if you'd like to replace a bunch of variables:

colNames = list(df1.columns)
colNames.remove('obs')
for variable in colNames:
    varDict = dict(df2.loc[df2['var']==variable][['value','label']].values)
    df1[variable+'_name'] = df1[variable].apply(lambda x: varDict[x])

For a million rows it takes about 1 second so should be reasonable fast.

Answer 2

Create a mapper dictionary from df2 using groupby

d = df2.groupby('Var').apply(lambda x: dict(zip(x['Value'], x['Label']))).to_dict()

{'gender': {1: 'male', 2: 'female'},
'job': {1: 'blue collar', 2: 'white collar'}}

Now map the values in df1 using outer key of the dictionary as column and inner dictionary is mapper

for col in df1.columns:
    if col in d.keys():
        df1[col] = df1[col].map(d[col])

You get

    obs gender  job
0   1   male    blue collar
1   2   male    white collar
2   3   female  white collar
3   4   male    blue collar