在 Python Pandas 中创建优先级

Question

I have a large dataset that shows every degree that an individual has and the year that it was obtained. Also, each individual has a corresponding ID. I am trying to find the year of birth of each individual using the year the degree was completed and the average age that degree is completed. The data set looks like the following:

For the average ages, I am assuming PhD is completed at 33, Master's at 30, and Bachelor's at 22.

person_id   degree                       degree_completion   year_of_birth
1           PhD                          2006                1973
1           BSc                          1999                1977
2           Ph.D.                        1995                1962
2           MBA                          2000                1970
2           B.A.                         1987                1965
3           Bachelor of Engineering      2005                1983
4           AB                           1997                1975
4           Doctor of Philosophy (PhD)   2003                1970                          

I have already created the system that calculates the year of birth of each individual, but I cannot figure how to create a priority system so that it picks the correct year of birth as there can be a different one calculated for each degree the individual has. I want the following prioritization: Bachelor's year of birth > PhD year of birth > Master's year of birth.

I have tried numerous things with the groupby function and the Categorial datatype. Also, there are hundreds of different forms the degrees are written in within the dataset so I have been depending on using regular expressions to both calculate the year of birth and create the prioritization system. This is what I have currently, but I cannot find a way to implement regex into this:

category1 = "^B[a-z]*|AB|A.B.|A.B|S.B."
category2 = "^P[a-z]*|Doctor of Philosophy[a-z]*"
category3 = "^M[a-z]*|Master[a-z]*"

file['edu_degree'] = pd.Categorical(file['edu_degree'], ordered=True, categories=[category1, category2, category3])

file.groupby('person_id')['edu_degree'].transform('max')

Also, this would be my desired output (year of births are replaced according to priority):

person_id   degree                       degree_completion   year_of_birth
1           PhD                          2006                1977
1           BSc                          1999                1977
2           Ph.D.                        1995                1965
2           MBA                          2000                1965
2           B.A.                         1987                1965
3           Bachelor of Engineering      2005                1983
4           AB                           1997                1975
4           Doctor of Philosophy (PhD)   2003                1975                          

Answer 1

Here's an idea, probably not the most elegant one (assumption is that your frame is named df ):

import re

re_category = re.compile(r"^(?:(Bachelor.*|BSc|B\.A\.|AB|A\.B\.|A\.B|S\.B\.)|"
                         + r"(PhD|Ph\.D\.|Doctor.*)|"
                         + r"(MBA|Master.*))$")
def category(match):
    for i, group in enumerate(match.group(1, 2, 3), start=1):
        if group:
            return str(i)

df['degree_cat'] = df.degree.str.strip().str.replace(re_category, category).astype(int)
df_year_of_birth = (df.sort_values('degree_cat').groupby('person_id', as_index=False)
                      .year_of_birth.first())
df = df.drop(columns=['degree_cat','year_of_birth']).merge(df_year_of_birth, on='person_id')

Some explanations:

Step 1 : A bit of reorganizing of your regex.

import re

re_category = re.compile(r"^(?:(Bachelor.*|BSc|B\.A\.|AB|A\.B\.|A\.B|S\.B\.)|"
                         + r"(PhD|Ph\.D\.|Doctor.*)|"
                         + r"(MBA|Master.*))$")

I've adjusted the patterns such that (1) the complete entry of column degree is matched, (2) included more possibities, and (3) escaped the . s. It's likely that you have to adjust it further! And I packed the categories in groups and concatenated them by | .

Step 2 : Creation of a degree_cat column (= category of the respective degree).

def category(match):
    for i, group in enumerate(match.group(1, 2, 3), start=1):
        if group:
            return str(i)

df['degree_cat'] = df.degree.str.strip().str.replace(re_category, category).astype(int)

I've used category as a repl -function which essentially replaces matches with their category. Check here how that works. The strip is just a precaution. The resulting column for your sample looks like:

0    2
1    1
2    2
3    3
4    1
5    1
6    1
7    2
Name: degree_cat, dtype: int64

Step 3 : Select the required years of birth.

df_year_of_birth = (df.sort_values('degree_cat').groupby('person_id', as_index=False)
                      .year_of_birth.first())

Here df gets sorted by the new column, grouped by person_id , and then the first item in year_of_birth is selected (which is the required year due to the sorting). Result for your sample:

   person_id  year_of_birth
0          1           1977
1          2           1965
2          3           1983
3          4           1975

Step 4 : Replace the values in year_of_birth with the requiered values.

df = df.drop(columns=['degree_cat','year_of_birth']).merge(df_year_of_birth, on='person_id')

Drop the old year_of_birth and the degree_cat column because they aren't needed any more, and the merge the df_year_of_birth dataframe on df along person_id to recreate the right year_of_birth column.

End result:

   person_id                      degree  degree_completion  year_of_birth
0          1                         PhD               2006           1977
1          1                         BSc               1999           1977
2          2                       Ph.D.               1995           1965
3          2                         MBA               2000           1965
4          2                        B.A.               1987           1965
5          3     Bachelor of Engineering               2005           1983
6          4                          AB               1997           1975
7          4  Doctor of Philosophy (PhD)               2003           1975

Answer 2

This is one possible solution, maybe not the most elegant one but still does its job

# define custom function to get the correct years
def find_best_year(df):

    cond1 = df['degree'].str.match(category1)
    cond2 = df['degree'].str.match(category2)
    cond3 = df['degree'].str.match(category3)
    
    if cond1.any():
        return df.loc[cond1, 'year_of_birth']
    elif cond2.any():
        return df.loc[cond2, 'year_of_birth']
    elif cond3.any():
        return df.loc[cond3, 'year_of_birth']
    else:
        raise ValueError("No condition was found.")


# create lookup table with best years
lookup_df = file\
    .groupby('person_id')\
    .apply(find_best_year)\
    .reset_index()\
    .drop(columns=['level_1'])
print(lookup_df)
#    person_id  year_of_birth
# 0          1           1977
# 1          2           1965
# 2          3           1983
# 3          4           1975


# desired output
file\
    .drop(columns=['year_of_birth'])\
    .merge(lookup_df, on='person_id', how='left')
#    person_id                      degree  degree_completion  year_of_birth
# 0          1                         PhD               2006           1977
# 1          1                         BSc               1999           1977
# 2          2                       Ph.D.               1995           1965
# 3          2                         MBA               2000           1965
# 4          2                        B.A.               1987           1965
# 5          3     Bachelor of Engineering               2005           1983
# 6          4                          AB               1997           1975
# 7          4  Doctor of Philosophy (PhD)               2003           1975

Answer 3

To apply the regex, you can make a function ( get_diploma ) to test them one after the other. Ideally in the order of the most probable ones (Bachelor first).

Then you can group by person_id and find the line with highest priority ( get_expected_age function).

import re

category1 = "^B[a-z]*|AB|A.B.|A.B|S.B."
category2 = "^P[a-z]*|Doctor of Philosophy[a-z]*"
category3 = "^M[a-z]*|Master[a-z]*"

diplomas = {category1: 'Bachelor', category2: 'PhD', category3: 'Master'}
ages = {'PhD': 33, 'Master': 30, 'Bachelor': 22}


def get_diploma(s):
    # for first matching regexp, return diploma
    for k in diplomas:
        if re.match(k, s):
            return diplomas[k]
    

        
df['degree_standardized'] = pd.Categorical(df['degree'].map(get_diploma),
                                           ordered=True,
                                           categories=['Master', 'PhD', 'Bachelor'])
# map the age from the standardized degree. NB. this could be fused with the previous step.
df['expected_age'] = df['degree_standardized'].map(ages)

def get_expected_age(d):
    # get degree with highest priority
    s = d.sort_values(by='degree_standardized').iloc[-1]
    d['year_of_birth'] = s['degree_completion']-s['expected_age']
    return d

df.groupby('person_id').apply(get_expected_age)

output:

   person_id                      degree  degree_completion  year_of_birth degree_standardized expected_age
0          1                         PhD               2006           1977                 PhD           33
1          1                         BSc               1999           1977            Bachelor           22
2          2                       Ph.D.               1995           1965                 PhD           33
3          2                         MBA               2000           1965              Master           30
4          2                        B.A.               1987           1965            Bachelor           22
5          3     Bachelor of Engineering               2005           1983            Bachelor           22
6          4                          AB               1997           1975            Bachelor           22
7          4  Doctor of Philosophy (PhD)               2003           1975                 PhD           33