在 Pandas 中合并两个大 dataframe

Question

How should i merge the label column in dataframe df (8 million rows) to another dataframe df2 (143 million rows) when the data size is that large?

Basically I just want to map the label column to df2 , all the data in df is included in df2 except the label column. Is there anyway I can solve this issue instead of using merge() ?

Tried to run the code below, but it keeps running for 5 hours but has no response.

result = pd.merge(df,df2,on=["X", "Y", "Z"], how='left')
result

df

df2

Answer 1

The are a few of obvious things I can see here that you can do:

1. Assuming you just want to add the label based on the X / Y / Z columns and R / G / B are superfluous, then drop the R / G / B columns of df as you don't need them in the final data frame, and you certainly don't need them being duplicated 143 million times.
2. Depending on how many unique values of X / Y / Z and their data type, you may be able to reduce the memory footprint by using categorical data types like so:

# Convert to categorical data types (if every value is unique, don't bother!)
for df_temp in [df, df2]:
    for col in ['X', 'Y', 'Z']:
        df_temp.loc[:, col] = df_temp[col].astype('category')
# Merge using less memory
result = pd.merge(df, df2, on=["X", "Y", "Z"], how='left')

3. Finally, you can try partitioning the data and doing a destructive conversion, you create several data frames each containing X in non-overlapping ranges and process them individually, then concatenate the individual results to give you the final result, eg:

result_dfs = []
ranges = [0, 1000, 2000, 3000, 4000, ...]
for start, end in zip(ranges[:-1], ranges[1:]):
    df_idx = (df['X'] >= start) & (df['X'] < end)
    df2_idx = (df2['X'] >= start) & (df2['X'] < end)
    result_dfs.append(
        pd.merge(
            df[df_idx], 
            df2[df2_idx], 
            on=["X", "Y", "Z"], 
            how='left'
        )
    )
    # Remove the original data to to reduce memory consumption
    df2 = df2[~df2_idx]
result = pd.concat(result_dfs)

This may still not work though, as you still need the full data set in memory twice for a short while when you do the final concatenation!

If none of these work, I'm afraid you need more memory, or you need to use something other than Pandas to solve your problem.