比较长度不相等的两个数据帧的行

Question

I'm fairly new to pandas and looking to get some feedback on how to tackle the problem below the best way. I'm trying to evaluate two column's values from two data frames of unequal lengths to find two cases:

1. id1 in data_set_2 doesn't exist in data_set_1.
2. (id1, id2) combination in data_set_2 does not exist in data_set_1.

The point of complexity is that I want to avoid using an apply or loop approach. These data sets can get absurdly large (examples below are intentionally simplified), and my understanding is that there can be more methodical ways to handle this.

data_set_1 = pd.DataFrame({"id1": ["A", "B", "C", "D"], "id2": ["1", "2", "2", "1"]})
data_set_2 = pd.DataFrame({"id1": ["A", "B", "F", "C", "D", "E"], "id2": ["1", "1", "2", "1", "1", "2"],"id3": ["1","2","3","4","5","6"]})

What I expect returned:

1. E, F

2.
(B, 1)
(F, 2)
(C, 1)
(E, 2)

What I've tried so far is the following:

To get products that do not exist in data_set_1:

data_set_2.loc[~(data_set_2.id1.isin(data_set_1.id1))] 

(This is where I'm not sure if this is the best way) - To get id1, id2 combinations that do not exist in data_set_1:

I tried an isin statement, it seemed like the lengths of the two dataframes appear to be an issue since pandas will evaluate for the same index row between the two dataframes AND it evaluates each columns independently.

I found that I could index multiple column values as such:

data_set_2.set_index(["id1", "id2"], inplace=True,drop=False)
data_set_1.set_index(["id1", "id2"], inplace=True,drop=False)

Which let's me do this:

~data_set_2[["id1","id2"]].isin(data_set_1)
A   1    False  False
B   1     True   True
F   2     True   True
C   1     True   True
D   1    False  False
E   2     True   True

Although this gives me what I want, i wasn't able to select the rows that evaluate to True in a loc selection operation:

data_set_2.loc[~data_set_2[["id1","id2"]].isin(data_set_1)]
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/local/bfm/lib/python/pandas/0.20.2-cp35/pandas/core/indexing.py", line 1328, in __getitem__
    return self._getitem_axis(key, axis=0)
  File "/usr/local/bfm/lib/python/pandas/0.20.2-cp35/pandas/core/indexing.py", line 1539, in _getitem_axis
    raise ValueError('Cannot index with multidimensional key')
ValueError: Cannot index with multidimensional key

Made me think that this isn't the right way to approach the problem. Any ideas on how this could be best achieved?

Answer 1

For your first case, you can use np.setdiff1d :

vals = np.setdiff1d(data_set_2.id1, data_set_1.id1)
print(vals)
array(['E', 'F'], dtype=object)

For the second case, setdiff1d does not work, but a simple set difference should do well enough.

vals = set(data_set_2.iloc[:, :2].apply(tuple, 1)) \
                       -  set(data_set_1.apply(tuple, 1))
print(vals)
{('B', '1'), ('C', '1'), ('E', '2'), ('F', '2')}

Alternatively, to improve upon your existing method, you might do something along these lines:

m = ~data_set_2[["id1","id2"]].isin(data_set_1)

print(m[m.all(1)])
          id1   id2
id1 id2
B   1    True  True
F   2    True  True
C   1    True  True
E   2    True  True

vals = m[m.all(1)].index.tolist()

print(vals)
[('B', '1'), ('F', '2'), ('C', '1'), ('E', '2')]

Answer 2

You can try to use anti-join to get data you want.

import pandas as pd
data_set_1 = pd.DataFrame({"id1": ["A", "B", "C", "D"], "id2": ["1", "2", "2", "1"]})
data_set_2 = pd.DataFrame({"id1": ["A", "B", "F", "C", "D", "E"], "id2": ["1", "1", "2", "1", "1", "2"],"id3": ["1","2","3","4","5","6"]})

# Merging two data frame on id1, then filtering base on indicator
data_result_1 = data_set_2.merge(data_set_1.loc[:, ["id1"]], on="id1", how="outer", indicator=True)
data_result_1 = data_result_1[data_result_1['_merge'] == 'left_only']

# Merging two data frame on id1 and id2, then filtering base on indicator
data_result_2 = data_set_2.merge(data_set_1.loc[:, ["id1", "id2"]], on=["id1", "id2"], how="outer", indicator=True)
data_result_2 = data_result_2[data_result_2['_merge'] == 'left_only']


print([tuple(x) for x in data_result_1.loc[:, ["id1"]].values])
print([tuple(x) for x in data_result_2.loc[:, ["id1", "id2"]].values])