在循环中创建多行 pandas.DataFrames 并附加到列表

Question

I'm very confused about this following behaviour: I have a loop, which creates simulated data based on a pandas.DataFrame . The output of each iteration is a new pandas.DataFrame with new columns ( x2 in the example below).

import pandas as pd
import random
mydf = pd.DataFrame({"x":[0]*2})

def addrand(x):
    return(x+random.normalvariate(0,1))

mysimulation = []
mycontrol = []
for i in range(0,5):
    mydf["x2"] = mydf["x"].apply(addrand)
    mydf["i"] = i
    mycontrol.append(i)
    mysimulation.append(mydf)
    
pd.concat(mysimulation)
#>    x        x2  i
0  0  1.023330  4
1  0 -0.428686  4
0  0  1.023330  4
1  0 -0.428686  4
0  0  1.023330  4
1  0 -0.428686  4
0  0  1.023330  4
1  0 -0.428686  4
0  0  1.023330  4
1  0 -0.428686  4

^{Created on 2020-09-08 by the reprexpy package}

What confuses me is: While the resulting list of pandas.DataFrames holds the expected amount of DataFrames rows (2 x 5 = 10), they are simply 5 copies of the last iteration. This is clearly visible from the id column. It should hold the numbers 0 to 4, but only contains the number 4. While on the hand, the list mycontrol behaves as expected and holds the numbers 0 to 4.

Why does this happen? And how can I resolve this?

Answer 1

• As you can see from the output below, mydf is updated with each iteration and added to mysimulation .
• However, with each iteration, you're doing an inplace update to mydf , and each mydf inside of mysimulation , is just a pointer, not a copy.
• The issue can be resolved by adding .copy() , like mysimulation.append(mydf.copy())

import random
import pandas as pd

random.seed(365)
def addrand(x):
    return(x+random.normalvariate(0,1))


mysimulation = []
mycontrol = []
display(mydf)  # display works in a jupyter notebook, otherwise use print
print('\n')
for i in range(0,5):
    print(i)

    mydf["x2"] = mydf["x"].apply(addrand)
    mydf["i"] = i
    display(mydf)

    mycontrol.append(i)
    mysimulation.append(mydf)
    display(mysimulation)
    print('\n')

With update

• mysimulation.append(mydf) to mysimulation.append(mydf.copy())

pd.concat(mysimulation)

[out]:
   x       x2  i
0  0  0.63265  0
1  0 -0.85868  0
0  0 -0.43199  1
1  0 -1.49446  1
0  0  0.23422  2
1  0 -0.74176  2
0  0  0.20195  3
1  0  1.61356  3
0  0  0.72138  4
1  0 -0.62529  4