检查 pandas 中的多值重复

Question

Inputs

I have a Dataframe with several columns.

proof_path = 

   #1  X  Y  #2 X_  Z  #3  W Z_  #4 W_ Y_
0  p1  a  b  p2  a  c  p2  a  c  p3  a  b
1  p1  a  b  p2  a  c  p3  a  c  p1  a  b
2  p1  a  b  p2  a  d  p3  e  d  p4  e  b

In the above Dataframe, I want to examine whether each row is duplicated between [#1, X, Y] , [#2, X_, Z] , [#3, W, Z_] , and [#4, W_, Y_] .

For example, in rows corresponding to index 0, [#2, X_, Z] and [#3, W_, Z_] overlap with [P2, a, c] . In addition, [#1, X, Y] and [#4, W_, Y_] in row corresponding to index 1 overlap [P1, a, b] . I'm going to drop rows that overlap between these multi-values from that dataframe.

My desired output is

output

proof_path = 

   #1  X  Y  #2 X_  Z  #3  W Z_  #4 W_ Y_
2  p1  a  b  p2  a  d  p3  e  d  p4  e  b

And i tried as follows.

triple_size = 3
for depth in range(int(len(proof_path.columns)/triple_size)-1):
    for i in range(1, int(len(proof_path.columns)/triple_size)-depth):
        current_rComp = proof_path.iloc[:, depth*size:(depth+1)*triple_size]
        next_rComp = proof_path.iloc[:, (depth+i)*size:(depth+i+1)*triple_size]
        current_rComp.columns = ['pred', 'subj', 'obj']
        next_rComp.columns = ['pred', 'subj', 'obj']
        proof_path = proof_path[current_rComp.ne(next_rComp).any(axis=1)]

Although these methods were able to achieve desired results, they are inefficient by generating sub set of proof_path for each iteration. Is there a simple way to accomplish these tasks?

Answer 1

To avoid the nested loops, you could use sets: For each row, put the four triples of values in a set. The number of elements in the set is the number of unique triples. Then you can use this number of unique triples as a mask to select rows:

import numpy as np

proof_path['n_unique_triples'] = \
    proof_path.apply(lambda row: len(set((tuple(row[0:3]),
                                          tuple(row[3:6]),
                                          tuple(row[6:9]),
                                          tuple(row[9:12])))), axis=1)
    
df_select = proof_path[proof_path.n_unique_triples == 4]
df_select

    #1  X   Y   #2  X_  Z   #3  W   Z_  #4  W_  Y_  n_unique_triples
2   p1  a   b   p2  a   d   p3  e   d   p4  e   b   4