如何清理列表，以及 pandas dataframe 中的元素列表？

Question

Edited:

After writing this:

m = df.explode('ID1').groupby('ID1')['ID2'].agg(list)

I have the following dataframe:

Ref         
45263     [['3105-BB', '3106-BB', '3201-BB', '3202-BB'],...
45256     [['3105-BB', '3106-BB', '3201-BB', '3202-BB'],...
48565     [['3159-CC', '3217-CC'], ['3159-CC', '3217-CC']]
49365     [['3159-CC', '3217-CC'], ['3159-CC', '3217-CC']]
47548     [['3107-CC', '3108-CC', '3201-CC', '3202-CC'],...

In col on the right, how do I remove the lists of list brackets, and the duplicates for each row. Ideally I'd like just a single list for each row?

eg for output:

Ref         
45263     ['3105-BB', '3106-BB', '3201-BB', '3202-BB']
45256     ['3105-BB', '3106-BB', '3201-BB', '3202-BB']
48565     ['3159-CC', '3217-CC']
49365     ['3159-CC', '3217-CC']
47548     ['3107-CC', '3108-CC', '3201-CC', '3202-CC']

Afterwards I will use m in the following:

df['ID4'] = df['Ref'].map(m)

This will return a final dataframe I am looking for.

Answer 1

Use set comprehension with flatten values of nested lists:

df['ID'] = df['ID'].apply(lambda x: list(set(z for y in x for z in y)))

If order is important use dict with keys trick:

df['ID'] = df['ID'].apply(lambda x: list(dict.fromkeys([z for y in x for z in y]).keys()))

If next processing is map, you need explode lists:

df = df.explode('ID').reset_index(drop=True)
print (df)
      Ref       ID
0   45263  3105-BB
1   45263  3106-BB
2   45263  3202-BB
3   45263  3201-BB
4   45256  3105-BB
5   45256  3106-BB
6   45256  3202-BB
7   45256  3201-BB
8   48565  3217-CC
9   48565  3159-CC
10  49365  3217-CC
11  49365  3159-CC
12  47548  3202-CC
13  47548  3108-CC
14  47548  3201-CC
15  47548  3107-CC

---

Sample :

df['ID1'] = df['ID'].apply(lambda x: list(set(z for y in x for z in y)))
df['ID2'] = df['ID'].apply(lambda x: list(dict.fromkeys([z for y in x for z in y]).keys()))
print (df)
     Ref                                        ID  \
0  45263    [[3105-BB, 3106-BB, 3201-BB, 3202-BB]]   
1  45256    [[3105-BB, 3106-BB, 3201-BB, 3202-BB]]   
2  48565  [[3159-CC, 3217-CC], [3159-CC, 3217-CC]]   
3  49365  [[3159-CC, 3217-CC], [3159-CC, 3217-CC]]   
4  47548    [[3107-CC, 3108-CC, 3201-CC, 3202-CC]]   

                                    ID1                                   ID2  
0  [3105-BB, 3106-BB, 3202-BB, 3201-BB]  [3105-BB, 3106-BB, 3201-BB, 3202-BB]  
1  [3105-BB, 3106-BB, 3202-BB, 3201-BB]  [3105-BB, 3106-BB, 3201-BB, 3202-BB]  
2                    [3217-CC, 3159-CC]                    [3159-CC, 3217-CC]  
3                    [3217-CC, 3159-CC]                    [3159-CC, 3217-CC]  
4  [3202-CC, 3108-CC, 3201-CC, 3107-CC]  [3107-CC, 3108-CC, 3201-CC, 3202-CC]  

EDIT:

f = lambda x: list(set(z for y in x for z in y)
df.explode('ID1').groupby('ID1')['ID2'].agg(f)