根据其他两个数据帧填充新的数据帧

Question

I want an efficient way to solve this problem below because my code seems inefficient.

First of all, let me provide a dummy dataset.

import numpy as np
import pandas as pd    
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"

df1= {'a0' : [1,2,2,1,3], 'a1' : [2,3,3,2,4], 'a2' : [3,4,4,3,5], 'a3' : [4,5,5,4,6], 'a4' : [5,6,6,5,7]}

df2 = {'b0' : [3,6,6,3,8], 'b1' : [6,8,8,6,9], 'b2' : [8,9,9,8,7], 'b3' : [9,7,7,9,2], 'b4' : [7,2,2,7,1]}

df1 = pd.DataFrame(df1)

df2 = pd.DataFrame(df2)

My actual dataset has more than 100,000 rows and 15 columns. Now, what I want to do is pretty complicated to explain, but here we go.

Goal: I want to create a new df using the two dfs above.

1) find the global min and max from df1. Since the value is sorted by row, column 'a' will always have the minimum each row, and 'e' will have the maximum. Therefore, I will find the minimum in column 'a0' and maximum in 'a4'.

Min = df1['a0'].min()
Max = df1['a4'].max()

Min
Max

2) Then I will create a data frame filled with 0s and columns of range(Min, Max). In this case, 1 through 7.

column = []
for i in np.arange(Min, Max+1):
    column.append(i)

newdf = pd.DataFrame(0, index = df1.index, columns=column)

3) The third step is to find the place where the values from df2 will go:

I wanna loop through each value in df1. and match each value with the column name in the new df in the same row.

For example, if we are looking at row 0 and go through each column; the values in this case [1,2,3,4,5]. Then the row 0 of the newdf, column 1,2,3,4,5 will be filled with the corresponding values from df2.

4) Lastly, each corresponding values in df2 (same place) will be added to the place where we found in step 2.

So, the very first row of the new df will look like this:

output = {'1' : [3], '2' : [6], '3' : [8], '4' : [9], '5' : [7], '6' : [0], '7' : [0]}

output = pd.DataFrame(output)

column 6 and 7 will not be updated because we didn't have 6 and 7 in the very first row of df1.

Here is my code for this process:

for rowidx in range(0, len(df1)):
    for columnidx in range(0,len(df1.columns)):
        new_column = df1[str(df1.columns[columnidx])][rowidx] 
        newdf.loc[newdf.index[rowidx], new_column] = df2['b' + df1.columns[columnidx][1:]][rowidx]

I think this does the job, but as I said, my actual dataset is huge with 2999999 rows and Min to Max range is 282 which means 282 columns in the new data frame.

So, the code above runs forever. If there is a faster way to do this. please help me with it. I think I learned something like map, reduce?? but I don't know if that applied here or there are other ways for this..

Thank you.

Answer 1

Idea is create default columns names in both DataFrame s, then concat of DataFrame.stack ed Series, add first 0 column to index, remove second level, so possible use DataFrame.unstack :

df1.columns = range(len(df1.columns))
df2.columns = range(len(df2.columns))

newdf = (pd.concat([df1.stack(), df2.stack()], axis=1)
           .set_index(0, append=True)
           .reset_index(level=1, drop=True)[1]
           .unstack(fill_value=0)
           .rename_axis(None, axis=1))
print (newdf)
   1  2  3  4  5  6  7
0  3  6  8  9  7  0  0
1  0  6  8  9  7  2  0
2  0  6  8  9  7  2  0
3  3  6  8  9  7  0  0
4  0  0  8  9  7  2  1

Another solutions:

comp =[pd.Series(a, index=df1.loc[i]) for i, a in enumerate(df2.values)]
df = pd.concat(comp, axis=1).T.fillna(0).astype(int)
print (df)
   1  2  3  4  5  6  7
0  3  6  8  9  7  0  0
1  0  6  8  9  7  2  0
2  0  6  8  9  7  2  0
3  3  6  8  9  7  0  0
4  0  0  8  9  7  2  1

Or:

comp = [dict(zip(x, y)) for x, y in zip(df1.values, df2.values)]
c = pd.DataFrame(comp).fillna(0).astype(int)
print (c)
   1  2  3  4  5  6  7
0  3  6  8  9  7  0  0
1  0  6  8  9  7  2  0
2  0  6  8  9  7  2  0
3  3  6  8  9  7  0  0
4  0  0  8  9  7  2  1