如何使用 Pandas Dataframe 对多个字符串进行分组和过滤？

Question

I'm a beginner for coding and I've tried to look for answers for a few days but I didn't succeed what I want to do so sorry in advance if it's easy or if it already exists somewhere... Let's say I have a df1 with columns: series_id and lesion_name and I would like to obtain a df2 by replacing the df1 with series_id and lung_ref_seg columns. I can have several lesion_name for the same series_id in df1 (left on the picture). In df2 (right on the picture) I would like to have only one lesion_name for one series_id original df1 and df2 wanted . If one series_id has a corresponding lesion_name which has 'lung' and 'PredCorr' I will take this as a first choice for lung_ref_seg. If lesion_name doesn't have 'lung' and 'PredCorr' but has 'lung' and 'From' I will take this as a second choice for lung_ref_seg. And if lesion_name doesn't have any of the two first choices I will take lesion_name which has 'Pred' and 'lung' as a third choice for lung_ref_seg. (lesion_name can have Nan values and I want to keep them in lung_ref_seg). I've tried a lot of things (groupby, filter, str.contains, isin, lambda row...) so I'll just put one code that I think is close to the solution:

lesion_name = test['lesion_name']
series_id = test['series_id']

def LungSegRef(lesion_name):
    for rows in series_id:
        if 'PredCorr' in lesion_name and 'lung' in lesion_name:
            return lesion_name
        elif 'PredCorr' not in lesion_name and 'From' in lesion_name and 'lung' in lesion_name:
            return lesion_name
        elif 'PredCorr' not in lesion_name and 'From' not in lesion_name and 'Pred' in lesion_name and 'lung' in lesion_name:
            return lesion_name
    return ''


# Apply the function RefLesionName 
test['lung_ref_seg'] = test['lesion_name'].apply(LungSegRef)

With this I don't have errors, I just have Nan values in the whole column lung_ref_seg and I still have multiple same series_id values. So I guess we could use groupby("series_id") somewhere and maybe my argument in the function is wrong. Thank you very much for your help !

The rows are not duplicates. I found something that should work but I'm struggling to differentiate values that contains Pred and PredCorr because when I use contains it doesn't make a difference between rows containing only Pred and rows containing PredCorr. With this code I can't use startswith() and endswith(). I'm trying to find answers with regex but for now I didn't find anything to differentiate rows that contains only Pred and lung compared to rows that contains PredCorr and lung.

 def select_row2(row2):
    if row2.lesion_name.str.contains("Pred" and "lung" and "Corr" and "From",na=True).any():
        return row2[row2.lesion_name.str.contains("Corr" and "lung",na=True)]
       
    elif row2.lesion_name.str.contains("Pred" and "lung" and "Corr",na=True).any():
        return row2[row2.lesion_name.str.contains("Corr" and "lung",na=True)]
    
    elif row2.lesion_name.str.contains("lung" and "Corr" and "From",na=True).any():
        return row2[row2.lesion_name.str.contains("Corr" and "lung",na=True)]
    
    elif row2.lesion_name.str.contains("Pred" and "lung" and "From",na=True).any():
        return row2[row2.lesion_name.str.contains("From" and "lung",na=True)]
    
    elif row2.lesion_name.str.contains("Pred" and "lung",na=True).any():
        return row2[row2.lesion_name.str.contains("Pred" and "lung",na=True)]
    
    elif row2.lesion_name.str.contains("lung" and "Corr",na=True).any():
        return row2[row2.lesion_name.str.contains("lung" and "Corr",na=True)]
    
    elif row2.lesion_name.str.contains("lung" and "From",na=True).any():
        return row2[row2.lesion_name.str.contains("lung" and "From",na=True)]
    
    else:
        return None 

test = test.groupby("series_id").apply(select_row2).reset_index(drop=True) 

Answer 1

I think you can approach this as a two step process:

1. First, filter down to the data that you want to keep (it seems like specific values containing specific strings are the one you want - but I'm a bit confused by your post here).
2. Second, drop duplicates from the series_id column. This will result in you just having one value for each of these.

As you mention, filtering using the df.filter function (though you can also do it as df['lung_ref_seg'] = df[df['lung_ref_seg'].str.contains('STRING TO KEEP')] .

Dropping duplicates can be done with the df.drop_duplicates(subset=['series_id'])

Answer 2

I found an answer that seems to work for now !

# We remove all lesion_name that contains string "tum" to work on creating the column lung_ref_seg (and we keep nan values)

test = test[~test.lesion_name.str.contains("tum",na=False)]

# Define the function to pick one lesion_name for one series_id prioritizing PredCorr first then From and then Pred for the last choice

def LungRefLesionName(row):
    if row.lesion_name.str.contains("(Pred)\w+" or "From" or "(Pred)\b", na=True).any():
        return row[row.lesion_name.str.contains("(Pred)\w+", na=True)]
        
    elif row.lesion_name.str.contains("From" or "(Pred)\b", na=True).any(): 
        return row[row.lesion_name.str.contains("From", na=True)] 
    
    elif row.lesion_name.str.contains("(Pred)\b" and "lung", na=True).any():
        return row[row.lesion_name.str.contains("(Pred)\b" and "lung", na=True)]

# Apply the function
test = test.groupby("series_id").apply(LungRefLesionName).reset_index(drop=True)


# Drop columns that we don't need anymore : segmentation_id, lesion_id, series_id and study_id

test = test.drop(['segmentation_id', 'lesion_id', 'series_id', 'study_id'], axis = 1)

# Renaming column lesion_name by lung_ref_lesion_name

test = test.rename(columns={"lesion_name": "lung_ref_lesion_name"})

I'll have modification to do with lesion_name containing 'tum' later so I guess I'll have to change some things but for now this code works for manipulate strings with 'lung' !