在python pandas中添加2个数据框

Question

I want to combine 2 seperate data frame of the following shape in Python Pandas:

Df1=
       A    B
    1  1    2
    2  3    4
    3  5    6

Df2 = 
       C    D
    1  a    b
    2  c    d
    3  e    f

I want to have as follows:

df = 
       A    B    C    D
   1   1    2    a    b
   2   3    4    c    d
   3   5    6    e    f

I am using the following code:

dat = df1.join(df2)

But problem is that, In my actual data frame there are more than 2 Million rows and for that it takes too long time and consumes huge memory.

Is there any way to do it faster and memory efficient?

Thank you in advance for helping.

Answer 1

If I've read your question correctly, your indexes align exactly and you just need to combine columns into a single DataFrame. If that's right then it turns out that copying over a column from one DataFrame to another is the fastest way to go ( [92] and [93] ). f is my DataFrame in the example below:

In [85]: len(f)
Out[86]: 343720

In [87]: a = f.loc[:, ['date_val', 'price']]
In [88]: b = f.loc[:, ['red_date', 'credit_spread']]

In [89]: %timeit c = pd.concat([a, b], axis=1)
100 loops, best of 3: 7.11 ms per loop

In [90]: %timeit c = pd.concat([a, b], axis=1, ignore_index=True)
100 loops, best of 3: 10.8 ms per loop

In [91]: %timeit c = a.join(b)
100 loops, best of 3: 6.47 ms per loop

In [92]: %timeit a['red_date'] = b['red_date']
1000 loops, best of 3: 1.17 ms per loop

In [93]: %timeit a['credit_spread'] = b['credit_spread']
1000 loops, best of 3: 1.16 ms per loop

I also tried to copy both columns at once but for some strange reason it was more than two times slower than copying each column individually.

In [94]: %timeit a[['red_date', 'credit_spread']] = b[['red_date', 'credit_spread']]
100 loops, best of 3: 5.09 ms per loop