如何串联pandas.DataFrames列

Question

I have a DataFrame called raw_df :

columns = ['force0', 'distance0', 'force1', 'distance1']

raw_data = [{'force0': 1.2, 'distance0': 0.0, 'force1': 0.5, 'distance1': 0.0},
            {'force0': 1.3, 'distance0': 0.1, 'force1': 0.6, 'distance1': 0.0},
            {'force0': 1.4, 'distance0': 0.2, 'force1': 0.7, 'distance1': 0.3},
            {'force0': 1.5, 'distance0': 0.5, 'force1': 0.8, 'distance1': 0.6}]

raw_df = pd.DataFrame(raw_data, columns=columns)

raw_df looks like this:

   force0  distance0  force1  distance1
0     1.2        0.0     0.5        0.0
1     1.3        0.1     0.6        0.0
2     1.4        0.2     0.7        0.3
3     1.5        0.5     0.8        0.6

At the moment there is no index but I would like the distance columns to be combined into one index so the columns are then:

          force0  force1
distance                
0.0          1.2     0.5
0.0          NaN.    0.6
0.1          1.3     NaN
0.2          1.4     NaN
0.3          NaN     0.7
0.5          1.5     NaN
0.6          NaN     0.8

Note that there were 2 entries in force1 for distance1 = 0.0.

The index (distances) should NOT be sorted: they increase then decrease variably and the original order for each test is important.

Stefan posted an amazing answer to my poorly-described question but it seemed to fill in any missing forces with other numbers (which would be misleading because there were no force measurements for those distances in those tests). I have used np.nan for missing values as I think this is what pandas does.

I think that merge or join might do what I need but couldn't understand the docs .

Perhaps pandas.DataFrame was not designed for such data, and I should use numpy.genfromtxt instead and just select the columns I need on the fly: I don't see any advantage to using a pandas.DataFrame if I'm selecting columns on the fly (because I'm not using an index in that case).

Thanks for any help.

Answer 1

If I'm understanding correctly, you are starting from a situation similar to this:

columns = list(sum(list(zip(['Forces{}'.format(i) for i in range(4)], ['Distances{}'.format(i) for i in range(4)])), ()))
df = pd.DataFrame(np.random.randint(1, 11, size=(100, 8)), columns=columns)

   Forces0  Distances0  Forces1  Distances1  Forces2  Distances2  Forces3  \
0        3           5        8           3        7           4        2   
1        1           4       10           9        9           3        6   
2       10           3        1           3        3           7        8   
3        2           1        3           6       10          10       10   
4        4           2        9           1        3          10        8   

   Distances3  
0           8  
1           5  
2           3  
3           8  
4           8  

and you are aiming to have the various Distance columns form a single index while the respective Force columns remain in place. You could columns remain in place. You could stack` the frame like so:

df.set_index([c for c in df.columns if c.startswith('Force')], inplace=True)
df = df.stack().reset_index(level=-1, drop=True).reset_index().rename(columns={0: 'Distance'})
df.set_index(['Distance'], inplace=True)

to get:

          Forces0  Forces1  Forces2  Forces3
Distance                                    
9               7        4        6        7
9               7        4        6        7
1               7        4        6        7
6               7        4        6        7
5               1        2        3        1

Answer 2

I solved the problem using a MultiIndex DataFrame :

1. Read each test into a separate DataFrame using pd.read_csv()
2. Combined the DataFrames into one using df = pd.concat(frame_list, keys=test_names)

Rather than write a long description here, I wrote a Jupyter notebook on the subject comparing the MultiIndex method against just keeping a standard Python list of DataFrames.