将 pandas.DataFrame 附加到另一个 pandas.DataFrame 的一列

Question

A similar question has already been asked here , however the exact answer to what the OP wanted was not provided. So, I will try again. I have data with 4 columns, named ['date', 'log time', 'choice', 'dt between ROI'] . I would now like to filter this data based on two criteria, here named ['LED', 'drum'] . In other words, if particular rows of the original pandas.DataFrame correspond to 'LED' , they get sorted under the 'LED' column of the master dataframe, and if they correspond to 'drum' , they get sorted under 'drum' column of the master dataframe. In this way, both 'LED' and 'drum' columns would have the same 4 subcolumns as the original data, ['date', 'log time', 'choice', 'dt between ROI'] . Additionally, the 'LED' and 'drum' columns would not necessarily have the same number of rows.

To start, I first created the master dataframe with the above described structure:

master_df = pandas.DataFrame({
    'distraction': ['LED','LED','LED','LED','drum','drum','drum','drum']),
    '': ['date', 'log time', 'choice', 'dt between ROI', 'date', 'log time', 'choice', 'dt between ROI']
})

master_df = master_df.set_index(['distraction', '']).transpose()

This resulted in the desired final structure:

In: master_df
Out:
distraction     LED                                             drum
                date    log time    choice    dt between ROI    date    log time    choice    dt between ROI

In: master_df['LED']
Out:
date    log time    choice    dt between ROI

Next, my filtering function returns certain rows of the original dataframe:

output = filter_function(original_df)

Hence, output has the same structure as original_df :

In: output
Out:
date    log time    choice    dt between ROI
x1      x2          x3        x4
y1      y1          y3        y4

Then I tried appending this output to the created master dataframe like so:

master_df['LED'] = master_df['LED'].append(output, ignore_index=True)

which resulted in the following error:

ValueError: Cannot set a frame with no defined index and a value that cannot be converted to a Series

Next, I tried:

master_df = master_df['LED'].append(output, ignore_index=True)

which simply overwrote the above created structure. What I really want to achieve is this:

In: master_df['LED'].append(output, ignore_index=True)
Out:
LED                                             drum
date    log time    choice    dt between ROI    date    log time    choice    dt between ROI
x1      x2          x3        x4
y1      y1          y3        y4

and likewise:

In: master_df['drum'].append(output, ignore_index=True)
Out:
LED                                             drum
date    log time    choice    dt between ROI    date    log time    choice    dt between ROI
                                                x1      x2          x3        x4
                                                y1      y1          y3        y4

I am not sure, if pandas can handle empty rows, but I guess NaN would be OK. After the filtering is done, I then wish to recall the two filtered datasets by simply calling master_df['LED'] or master_df['drum'] . Is there a way to do this?

Many thanks for your help!

EDIT: Fixed criterium -> distraction to avoid confusion.

Answer 1

The point of @Dani is that your code does not work. Anyway it works if one replaces 'criterium' with 'distraction' which I assume was the intent.

So to your question, you can do the following. You can prepend your output with another level of column multi-index so that it matches the column structure of master_df . Then you can safely append or concat

# if this goes into LED group; otherwise use 'drum' 
output_LED = pd.concat([output], keys = ['LED'], axis=1)
master_df2 = master_df.append(output_LED)
master_df2

produces

    LED drum
choice  date    dt between ROI  log time    choice  date    dt between ROI  log time
0   x3  x1  x4  x2  NaN NaN NaN NaN
1   y3  y1  y4  y1  NaN NaN NaN NaN

and

masket_df2['LED']

produces

choice  date    dt between ROI  log time
0   x3  x1  x4  x2
1   y3  y1  y4  y1

and

master_df2['drum']

produces

choice  date    dt between ROI  log time
0   NaN NaN NaN NaN
1   NaN NaN NaN NaN