Pandas 数据框合并行

Question

My problem is a large data frame which I would like to clear out. The two main problems for me are:

1. The whole data frame is time-based. That means I can not shift rows around, otherwise, the timestamp wouldn't fit anymore.

2. The data is not always in the same order.

Here is an example to clarify

index  a  b  c  d  x1  x2  y1  y2  t
0                  1   2           0.2
1      1  2                        0.4
2                          2   4   0.6
3                  1   2           1.8
4                          2   3   2.0
5                  1   2           3.8
6                          2   3   4.0
7            2  5                  4.2

The result should be looking like this

index  a  b  c  d  x1  x2  y1  y2  t
0                  1   2   2   4   0.2
1      1  2                        0.4
3                  1   2   2   3   1.8
5                  1   2   2   3   3.8
7            2  5                  4.2

This means I would like, to sum up, the right half of the df and keep the timestamp of the first entry. The second problem is, there might be different data from the left half of the df in between.

Answer 1

This may not be the most general solution, but it solves your problem:

First , isolate the right half:

r = df[['x1', 'x2', 'y1', 'y2']].dropna(how='all')

Second , use dropna applied column by column to compress the data:

r_compressed = r.apply(
    lambda g: g.dropna().reset_index(drop=True),
    axis=0
).set_index(r.index[::2])

You need to drop the index otherwise pandas will attempt to realign the data. The original index is reapplied at the end (but only with every second index label) to facilitate reinsertion of the left half and the t column.

Output (note the index values):

    x1   x2   y1   y2
0  1.0  2.0  2.0  4.0
3  1.0  2.0  2.0  3.0
5  1.0  2.0  2.0  3.0

Third , isolate left half:

l = df[['a', 'b', 'c', 'd']].dropna(how='all')

Fourth , incorporate the left half and t column to compressed right half:

out = r_compressed.combine_first(l)
out['t'] = df['t']

Output:

     a    b    c    d   x1   x2   y1   y2    t
0  NaN  NaN  NaN  NaN  1.0  2.0  2.0  4.0  0.2
1  1.0  2.0  NaN  NaN  NaN  NaN  NaN  NaN  0.4
3  NaN  NaN  NaN  NaN  1.0  2.0  2.0  3.0  1.8
5  NaN  NaN  NaN  NaN  1.0  2.0  2.0  3.0  3.8
7  NaN  NaN  2.0  5.0  NaN  NaN  NaN  NaN  4.2