在熊猫中追加数据框

Question

I have a 'for' loop that is calling a function (y) on each iteration. The function returns a 5 column by ten row dataframe called phstab.

for j in cycles
    phstab=y(j)

The last column in the dataframe is the only one that changes. The value in the last column is the value for cycles. All the other values in the other columns all stay the same on each iteration. So if the loop iterates for time for example, it will produce four separate instances of phstab; each instance with a different value of cycles.

I'd like to append phstab on each iteration so so the output is just one long dataframe instead of four instances. I tried inserting the following statement in the loop but it didn't work

phstab=phstab.append(phstab)

How do I get one single dataframe instead of four separate instances ?

Answer 1

I'm assuming your y(j) returns something like this:

In [35]: def y(j):
    ...:     return pd.DataFrame({'a': range(10), 
    ...:                          'b': range(10), 
    ...:                          'c': range(10), 
    ...:                          'd': range(10), 
    ...:                          'e_cycle' : j})

To iterate over this function, adding columns for each iterations, I'd do something like this. On the first pass, the dataframe is just set to phstab. On each subsequent iteration, a new column is added to phstab based on results of y(j).

I'm assuming you need to rename columns, if y(j) returns a unique column based on the value of j, you'll have to modify to fit.

In [38]: cycles = range(5)

In [38]: for i,j in enumerate(cycles):
    ...:     if i == 0:
    ...:         phstab = y(j)
    ...:         phstab = phstab.rename(columns = {'e_cycle' : 'e_' + str(j)})
    ...:     else:
    ...:         phstab['e_' + str(j)] = y(j)['e_cycle']

In [38]: phstab
Out[38]: 
   a  b  c  d  e_0  e_1  e_2  e_3  e_4
0  0  0  0  0    0    1    2    3    4
1  1  1  1  1    0    1    2    3    4
2  2  2  2  2    0    1    2    3    4
3  3  3  3  3    0    1    2    3    4
4  4  4  4  4    0    1    2    3    4
5  5  5  5  5    0    1    2    3    4
6  6  6  6  6    0    1    2    3    4
7  7  7  7  7    0    1    2    3    4
8  8  8  8  8    0    1    2    3    4
9  9  9  9  9    0    1    2    3    4

[10 rows x 9 columns]

Edit: Thanks for clarifying. To have the output in long format, you can you pd.concat, as below.

In [47]: pd.concat([y(j) for j in cycles], ignore_index=True)
Out[47]: 
    a  b  c  d  e_cycle
0   0  0  0  0        0
1   1  1  1  1        0
2   2  2  2  2        0
3   3  3  3  3        0
4   4  4  4  4        0
5   5  5  5  5        0
6   6  6  6  6        0
7   7  7  7  7        0
8   8  8  8  8        0
9   9  9  9  9        0
10  0  0  0  0        1
11  1  1  1  1        1
.....

[50 rows x 5 columns]

Answer 2

I believe a very simple solution is

my_dataframes = []
for j in cycles:
    phstab = y(j)
    my_dataframes.append(phstab)
full_dataframe = pd.concat(my_dataframes)

Alternatively and more concisely (credit to @chrisb above)

full_dataframe = pd.concat([y(j) for j in cycles], ignore_index=True)

pd.concat merges a list of dataframes together vertically. Ignoring the index is important so that the merged version doesn't retain the indices of the individual dataframes - otherwise you might end up with an index of [0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3] when instead you'd want [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15].