Pandas 删除两列的反向重复

Question

An example DataFrame:

df = pd.DataFrame({'node_a': ['X', 'X', 'X', 'Y', 'Y', 'Y', 'Z', 'Z', 'Z'],
                   'node_b': ['X', 'Y', 'Z', 'X', 'Y', 'Z', 'X', 'Y', 'Z'],
                   'value':  [  2,   8,   1,   8,   7,   3,   1,   3,   2]})

    node_a  node_b  value
0   X       X       2
1   X       Y       8
2   X       Z       1
3   Y       X       8
4   Y       Y       7
5   Y       Z       3
6   Z       X       1
7   Z       Y       3
8   Z       Z       2

I need to remove reversed duplicates, eg keep node_a = 'X', node_b = 'Y' but remove node_a = 'Y', node_b = 'X'.

Desired output:

    node_a  node_b  value
0   X       X       2
1   X       Y       8
2   X       Z       1
4   Y       Y       7
5   Y       Z       3
8   Z       Z       2

Please note I need a general solution not specific to this actual data.

Answer 1

Let's use np.sort along axis=1 to sort node_a and node_b and assign these sorted columns to the dataframe then use drop_duplicates on the dataframe to drop the duplicate entries in dataframe based on these assigned columns:

df[['x', 'y']] = np.sort(df[['node_a', 'node_b']], axis=1)
out = df.drop_duplicates(['x', 'y']).drop(['x', 'y'], 1)

Result:

print(out)
  node_a node_b  value
0      X      X      2
1      X      Y      8
2      X      Z      1
4      Y      Y      7
5      Y      Z      3
8      Z      Z      2

Answer 2

You could do the following:

# duplicates regardless the order
un_dups = pd.Series([frozenset(row) for row in df[['node_a', 'node_b']].to_numpy()]).duplicated()

# duplicates with the same order
o_dups = df.duplicated(subset=['node_a', 'node_b'])

# keep only those that are not duplicates with reverse order xor
mask = ~(un_dups ^ o_dups)

print(df[mask])

Output

  node_a node_b  value
0      X      X      2
1      X      Y      8
2      X      Z      1
4      Y      Y      7
5      Y      Z      3
8      Z      Z      2

The idea is to create a mask that will be False if you are a duplicate in reverse order.

To better understand the approach see the truth values:

  node_a node_b  value  un_dups  o_dups    xor
0      X      X      2    False   False  False
1      X      Y      8    False   False  False
2      X      Z      1    False   False  False
3      Y      X      8     True   False   True
4      Y      Y      7    False   False  False
5      Y      Z      3    False   False  False
6      Z      X      1     True   False   True
7      Z      Y      3     True   False   True
8      Z      Z      2    False   False  False

As you can see the xor ( exclusive or ) shows that it output is true whenever the inputs differ. Given that an ordered duplicated is going to be also duplicated when unordered, then xor is only true when the values in the column are duplicates in reverse order.

Finally notice that the mask is the negation of the xor, ie those values that are not duplicates.

Answer 3

Here's one way to do it which involves creating a new temporary column that will sort the order of node_a and node_b in each row, and then drop duplicates, keeping the first instance of the ordering:

df['sorted'] = df.apply(lambda x: ''.join(sorted([x['node_a'],x['node_b']])),axis=1)

#   node_a node_b  value sorted
# 0      X      X      2     XX
# 1      X      Y      8     XY
# 2      X      Z      1     XZ
# 3      Y      X      8     XY
# 4      Y      Y      7     YY
# 5      Y      Z      3     YZ
# 6      Z      X      1     XZ
# 7      Z      Y      3     YZ
# 8      Z      Z      2     ZZ

df.drop_duplicates(subset='sorted').drop('sorted',axis=1)

#   node_a node_b  value
# 0      X      X      2
# 1      X      Y      8
# 2      X      Z      1
# 4      Y      Y      7
# 5      Y      Z      3
# 8      Z      Z      2