Selecting specific rows in pandas dataframe merging

Question

I have 4 dataframes in the array that I keep.

     0    1    2
0  0.0  1.0  2.0
1  0.0  1.0  2.0
2  0.0  1.0  2.0
3  0.0  1.0  2.0
4  0.0  2.0  3.0
5  0.0  2.0  3.0
6  0.0  3.0  4.0
7  0.0  3.0  4.0

     0    1    2
0  1.0  4.0  4.0
1  1.0  5.0  5.0

     0    1    2
0  2.0  6.0  4.0

     0    1    2
0  3.0  7.0  6.0

Roughly what I want to do is combine these dataframes according to certain conditions. For example, adding data frames starting with the value in column 1 in row 0 in the first data frame to the first data frame. As a result, the first dataframe takes a format like this:

     0    1    2   3    4    5
0  0.0  1.0  2.0
1  0.0  1.0  2.0
2  0.0  1.0  2.0 1.0  4.0  4.0
3  0.0  1.0  2.0 1.0  5.0  5.0
4  0.0  2.0  3.0
5  0.0  2.0  3.0 2.0  6.0  4.0
6  0.0  3.0  4.0
7  0.0  3.0  4.0 3.0  7.0  6.0

The state of the data in dict format.

{0: {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 4: 0.0, 5: 0.0, 6: 0.0, 7: 0.0}, 1: {0: 1.0, 1: 1.0, 2: 1.0, 3: 1.0, 4: 2.0, 5: 2.0, 6: 3.0, 7: 3.0}, 2: {0: 2.0, 1: 2.0, 2: 2.0, 3: 2.0, 4: 3.0, 5: 3.0, 6: 4.0, 7: 4.0}}
{0: {0: 1.0, 1: 1.0}, 1: {0: 4.0, 1: 5.0}, 2: {0: 4.0, 1: 5.0}}
{0: {0: 2.0}, 1: {0: 6.0}, 2: {0: 4.0}}
{0: {0: 3.0}, 1: {0: 7.0}, 2: {0: 6.0}}

I also tried various algorithms and could not solve the problem in this part.

Answer 1

If the rows where 1 stays the same on the first dataframe, you can play a bit with the indexes, let't create the dataframes:

df = pd.DataFrame({0: {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 4: 0.0, 5: 0.0, 6: 0.0, 7: 0.0}, 1: {0: 1.0, 1: 1.0, 2: 1.0, 3: 1.0, 4: 2.0, 5: 2.0, 6: 3.0, 7: 3.0}, 2: {0: 2.0, 1: 2.0, 2: 2.0, 3: 2.0, 4: 3.0, 5: 3.0, 6: 4.0, 7: 4.0}})
df_1 = pd.DataFrame({0: {0: 1.0, 1: 1.0}, 1: {0: 4.0, 1: 5.0}, 2: {0: 4.0, 1: 5.0}})
df_2 = pd.DataFrame({0: {0: 2.0}, 1: {0: 6.0}, 2: {0: 4.0}})
df_3 = pd.DataFrame({0: {0: 3.0}, 1: {0: 7.0}, 2: {0: 6.0}})

Rename columns of df_X:

df_1.columns = [4, 5, 6]
df_2.columns = [4, 5, 6]
df_3.columns = [4, 5, 6]

And then: change index of df using cumcount , then merge dataframe using index and 1 as keys, finally, adjust your dataframe with iloc and moving nan values to the thop

df.index = df.groupby(1).cumcount()

df_merged = df.reset_index().merge(
    pd.concat([df_1, df_2, df_3], axis=0).reset_index(),
    left_on=(1, 'index'),
    right_on=(4, 'index'),
    how='left'
).drop('index', axis=1)

df_merged.groupby(1).apply(
    lambda df: df.iloc[
        np.roll(np.arange(df.shape[0]), df[4].isnull().sum())
    ]
).reset_index(drop=True)


#       0       1       2       4       5       6
# 0     0.0     1.0     2.0     NaN     NaN     NaN
# 1     0.0     1.0     2.0     NaN     NaN     NaN
# 2     0.0     1.0     2.0     1.0     4.0     4.0
# 3     0.0     1.0     2.0     1.0     5.0     5.0
# 4     0.0     2.0     3.0     NaN     NaN     NaN
# 5     0.0     2.0     3.0     2.0     6.0     4.0
# 6     0.0     3.0     4.0     NaN     NaN     NaN
# 7     0.0     3.0     4.0     3.0     7.0     6.0

Answer 2

First, rename the columns of df2 , df3 , df4 from 0,1,2 to 3,4,5

for df in [df2, df3, df4]:
    df.rename(columns={0:3, 1:4, 2:5}, inplace=True)

Second, change the index of these columns to the row index where you want to append them in df1

df2.index = [2,3]
df3.index = [5]
df4.index = [7]

Now, you can use two consecutive pd.concat , to have the expected dataframe output. The first concat concatenates df2 , df3 , and df4 in the row, second concat concatenates this output to df1 on column, something like this:

pd.concat([df1,pd.concat([df2, df3, df4], axis=0)], axis=1 )

OUTPUT

     0    1    2    3    4    5
0  0.0  1.0  2.0  NaN  NaN  NaN
1  0.0  1.0  2.0  NaN  NaN  NaN
2  0.0  1.0  2.0  1.0  4.0  4.0
3  0.0  1.0  2.0  1.0  5.0  5.0
4  0.0  2.0  3.0  NaN  NaN  NaN
5  0.0  2.0  3.0  2.0  6.0  4.0
6  0.0  3.0  4.0  NaN  NaN  NaN
7  0.0  3.0  4.0  3.0  7.0  6.0