给定两个列表，在 dataframe 的列中查找共现？

Question

I am trying to clean a dataset in such a way that I want to find the co-occurrence of two strings coming from two separate lists in the columns of a dataframe, in order to obtain the frequency that those two events co-occur together.

My first list has a length of 27 as the following:

df_dis = ['heart attack', 'panic disorder', 'bowel cancer' ...] 

And my second list has a length of 57 as the follows:

df_sym = ['chest pain', 'weight loss', 'extreme hand movement'...]

My dataframe (df) is made up of 5 columns as follows, ( I am only showing the first 5 rows):

    Diseases    Symptoms   Counts   Disease_str                  Symptoms_str
0   4464711     4831330     5289    heart attack                 chest pain
1   4147316     4402204     374     bowel obstructive cancer     weight loss
2   4317917     4317917     510     panic disorder               weight loss
3   4012264     5046090     1154    COPD                         panic attack
4   4819042     5136449     121     heart attack                 memory loss

The shape of this df is (18518404, 5). This df will contain repeats of the events in both lists but also, they may contain one, two, all the words or additional words, so I am trying to pick up as many of those words (using the lists) to find how many times the events co-occur.

What I did next to find the co-occurring events, I iterated over the dataframe's columns, Disease_str and Symptoms_str given the two lists, to get the.value_counts() from the Counts columns, as follows:

for i, j in map(df_dis, df_sys):
    val_counts_ = df['Counts'][(df['Disease_str'] == df_dis[i]) & (df['Symptoms_str'] == df_sys[j])].value_counts()

I am using the operand &, because I want the intersection rather than the union |.

However, I get an error message:

TypeError: 'list' object is not callable

I have also tried zip(df_dis, df_sys) but instead, I still get an error message. This time it is a TypeError: list indices must be integers or slices, not str .

What I would like to obtain is a csv file, that shows the combination of i & j in one column, the counts, and the total from the number of times i & j co-occurred?

I would appreciate any help and since I am new to programming and pandas, I would also appreciate any explanations so I can jot them down in my notebook so I can try to understand them better.

Thank you for the help.

Answer 1

you can create a mask where the column Disease_str isin the list df_dys and same with Symptoms_str column. Then you filter the rows with this mask, you groupby the two columns and agg on the column Counts to get the count and the sum . Now to get all the possible combinations from your two lists, you can reindex with the MultiIndex.from_product of the two lists.

m = df['Disease_str'].isin(df_dis) & df['Symptoms_str'].isin(df_sym)
df_ = (df[m].groupby(['Disease_str', 'Symptoms_str'])
            ['Counts'].agg(['count','sum']) #or just ['Counts'].size() if you don't care of the sum
            .reindex(pd.MultiIndex.from_product([df_dis, df_sym], 
                                                names=['Disease_str', 'Symptoms_str']), 
                     fill_value=0)
            .reset_index()
      )
print (df_)
      Disease_str           Symptoms_str  count   sum
0    heart attack             chest pain      1  5289
1    heart attack            weight loss      0     0
2    heart attack  extreme hand movement      0     0
3  panic disorder             chest pain      0     0
4  panic disorder            weight loss      1   510
5  panic disorder  extreme hand movement      0     0
6    bowel cancer             chest pain      0     0
7    bowel cancer            weight loss      0     0
8    bowel cancer  extreme hand movement      0     0