如何遍历两个不同大小的数据帧？

Question

Specifically I want to iterate through two dataframes, one being large and one being small.

Ultimately, I would like to compare values within a certain column.

I tried creating a nested for loop; the outer loop iterating through the large dataframe and the inner loop iterating through the small dataframe however I am having difficulties.

I'm looking for a way to identify that the "name" and "value" in my large dataframe that matches my small dataframe.

Background info: I am using the panda library.

Large dataframe:

Small dataframe:

Name     Value
SF       12.84
TH      -49.45

Answer 1

If the goal is to iterate through one, or especially more, DataFrame s, then explicit for loops is usually the wrong move. In this case, because you're trying to

identify that the "name" and "value" in my large dataframe that matches my small dataframe,

the operation that you're looking for is either pd.merge or pd.DataFrame.join which do the comparisons "under the hood" and return matching information. So, say you have the 2 DataFrame s and they're called large and small . Then

import pandas as pd
new_large = pd.merge(left=large,
                     right=small,
                     how='left',
                     on=('Name', 'Value'),
                     indicator=True)

new_large._merge = new_large._merge.apply(lambda x: 1 if x=='both' else 0)

By doing a left join between large and small ( how='left' ), pd.merge returns the rows in large that contain a match in small on the ('Name', 'Value') tuple. Then, most of the heavy lifting is done by the indicator keyword that, quoting the pd.merge version 0.25.0 docs:

If True, adds a column to output DataFrame called "_merge" with information on the source of each row. Information column is Categorical-type and takes on a value of "left_only" for observations whose merge key only appears in 'left' DataFrame, "right_only" for observations whose merge key only appears in 'right' DataFrame, and "both" if the observation's merge key is found in both.

So, new_large is the original large DataFrame with a new column called _merge the entries of which correspond to the rows of large that matched small just on Name (by the value 'left_only' ) and the rows that matched on Name as well as Value ; the latter having the value both . The last step is changing both and left_only to 1 and 0 , as you specified.

Now, the left join returned what it did because both of the Name values in the small DataFrame were present in the large DataFrame so the left-join of large and small returned the whole large DataFrame . When this is not the case, there will be pd.NaN values resulting from pd.merge and you'll have to employ a few more tricks to get the nice Boolean (integer) column to show what matched and what didn't. HTH.