Replace string in the same column in Series pandas

Question

Things could be not easy to understand at first sight, but step by step...

This is beginning of my circa 50000 rows Dataframe df_answers_clean

After using len(set(df_answers_clean['Race'])) I get 98 unique positions, which is too much for my future classifications. I made a example - list of the 25 positions of them are below:

['Native American, Pacific Islander, or Indigenous Australian; South Asian; White or of European descent', 'Hispanic or Latino/Latina; South Asian', 'East Asian; Hispanic or Latino/Latina',
 'East Asian', 'Black or of African descent; East Asian; South Asian; White or of European descent',
 'Black or of African descent; East Asian; Hispanic or Latino/Latina; Middle Eastern; Native American, Pacific Islander, or Indigenous Australian; South Asian; White or of European descent',
 'Hispanic or Latino/Latina; White or of European descent', 'White or of European descent; I prefer not to say', 'South Asian; White or of European descent', 'White or of European descent',
 'Hispanic or Latino/Latina', 'Black or of African descent; I don’t know; I prefer not to say',
 'Native American, Pacific Islander, or Indigenous Australian; White or of European descent; I don’t know', 'East Asian; White or of European descent; I don’t know', 'Native American, Pacific Islander, or Indigenous Australian', 'South Asian; White or of European descent; I don’t know',
 'Black or of African descent; Middle Eastern; White or of European descent; I don’t know',
 'Hispanic or Latino/Latina; Middle Eastern; White or of European descent',
 'Middle Eastern; White or of European descent',
 'Middle Eastern; South Asian']

I clean this mess with many lines of code:

df_answers_clean['Race'] = df_answers_clean['Race'].str.replace('^Black or of African descent[\s\S]*', 'Black or of African descent')    
df_answers_clean['Race'] = df_answers_clean['Race'].str.replace('^East Asian[\s\S]*', 'East Asian')    
df_answers_clean['Race'] = df_answers_clean['Race'].str.replace('^Hispanic or Latino/Latina[\s\S]*', 'Hispanic or Latino/Latina')
df_answers_clean['Race'] = df_answers_clean['Race'].str.replace('^Middle Eastern[\s\S]*', 'Middle Eastern')
df_answers_clean['Race'] = df_answers_clean['Race'].str.replace('^Native American, Pacific Islander, or Indigenous Australian[\s\S]*', 'Native American, Pacific Islander, or Indigenous Australian')    
df_answers_clean['Race'] = df_answers_clean['Race'].str.replace('^South Asian[\s\S]*', 'South Asian')    
df_answers_clean['Race'] = df_answers_clean['Race'].str.replace('^White or of European descent[\s\S]*', 'White or of European descent')    
df_answers_clean['Race'] = df_answers_clean['Race'].str.replace('^I don’t know[\s\S]*', 'No data')    
df_answers_clean['Race'] = df_answers_clean['Race'].str.replace('^I prefer not to say[\s\S]*', 'No data')

The result is unique group which only now is useful for the later classification task:

{'Black or of African descent',
 'East Asian',
 'Hispanic or Latino/Latina',
 'Middle Eastern',
 'Native American, Pacific Islander, or Indigenous Australian',
 'No data',
 'South Asian',
 'White or of European descent'}

As I said - it works, but many lines of duplicated code is not functional/practical.

My another idea of doing this was make a list of my final result ( race_names_change ) and put everything by for-loop:

race_names_change = ['Black or of African descent', 'East Asian', 'Hispanic or Latino/Latina', 'Middle Eastern', 'South Asian', 'Native American, Pacific Islander, or Indigenous Australian', 'White or of European descent']

for i in race_names_change:
    replace_string = str('^'+ i +'[\s\S]*')
    df_answers_clean['Race'].str.replace('replace_string', i, regex=True)

But unfortunately it does not work - list is the same as at the beginning (98 positions).

Maybe is something wrong in the loop code or any other way of doing this (map, apply)?

Thanks for advice.

Answer 1

If you can create a dictionary with the required regex patterns and corresponding outputs, then you can simply use pd.Series.replace()

d = {
    'pattern1':'output1',
    'pattern2':'output2'
    }

df_answers_clean['Race'].replace(d, regex=True)

Note, pd.Series.str.replace() is different than pd.Series.replace()

---

Try this -

race_names_change = ['Black or of African descent', 'East Asian', 'Hispanic or Latino/Latina', 'Middle Eastern', 'South Asian', 'Native American, Pacific Islander, or Indigenous Australian', 'White or of European descent']

d = {}
for i in race_names_change:
    replace_string = str('^'+ i +'[\s\S]*')
    replace_string
    d.update({replace_string:i})
    
df_answers_clean['Race'].replace(d, regex=True)