如何有效地更改熊猫数据框值

Question

Say I have 2 dataframes.

first :

100K rows
columns: ID, ch1,ch2...ch10,
binary values

second

1000K rows
columns: ID, CH.

say I wanna add to DF2 the values of DF1, in 1 column that it will fit the ID&chan.

I tried to do it with a simple for-loop but it took a few minutes just to get past the first 10K rows.. I wonder how it can be done effectively - as if I've done it with numpy, it'd work faster. just to clarify, the IDs can be shuffled in DF2.. so I can't predict it's location based on DF1.

thanks in advance!

Answer 1

Updated post:

Ah, okay. I think I get what you're saying. Maybe try something like this out:

df1 = pd.DataFrame({ "ID":[1,2,3], "ch1":[0,-1,0], "ch2":[0,0,0], "ch3":[-1,0,1] })

new_df = pd.DataFrame()

min_ch = 1
max_ch = 3
for i in range(min_ch,max_ch+1):
    this_ch_str = "ch"+str(i)
    temp_df = df1[["ID",this_ch_str]].copy()
    temp_df["CH"] = i
    temp_df = temp_df.rename(columns={this_ch_str:"val"})
    temp_df = temp_df[["CH","ID","val"]]
    new_df = new_df.append(temp_df)

It's a bit convoluted, but it gets the job done.

Original post:

I think the "merge" function might be what you're looking for.

Here's a quick example:

import pandas as pd
df1 = pd.DataFrame({"ID":[1,2,3,4,5],"col1":["A","B","C","D","E"]})
df2 = pd.DataFrame({"ID":[1,2,3,4,5,6,7,8,9,10],"col2":["x","y","z","x","y","z","x","y","z","x"]})
df3 = pd.merge(left=df1,right=df2,on="ID",how="left")

The result stored in df3 will look like this:

   ID col1 col2
0   1    A    x
1   2    B    y
2   3    C    z
3   4    D    x
4   5    E    y

Hope that's what you were looking for!