Fill pandas Dataframe columns of different length with a loop

Question

This is an example of what I would like to do:

history=Dataframe()

range= 40
for k in range (range):

    epochs=k+1
    loss=np.random.rand(k+1,1)


history[k]=loss

This fills the Dataframe with k columns, with the loss array.

The problem is that, for the following iteration, loss is bigger than in the previous iteration, and bigger in the following.

So, there is a conflict between the dataframe index:

raise ValueError('Length of values does not match length of ' 'index')
ValueError: Length of values does not match length of index

Is there any way to fill the dataframe as I want without this problem?

If the dataframe wasn't filled with a loop, I would think of filling it with NaNs.

Answer 1

Is this what you want ?

l= 3
for k in range(l):

    epochs=k+1
    loss=np.random.rand(k+1,1)
    if k ==0:
        history=pd.DataFrame(loss)
    else:
        history=pd.concat([history,pd.DataFrame(loss)],axis=1)
history.columns=range(3)

history
          0         1         2
0  0.043321  0.882806  0.578819
1       NaN  0.216240  0.558565
2       NaN       NaN  0.739184