我如何在 pandas/dataframe 中执行下面的 python 操作

Question

below is my dataframe

df = pd.DataFrame({
                   'Year': [2021, 2022, 2023, 2024, 2025],
                    'Tval' : [1, 9, 8, 1, 6]
})

I want to create a new column with output as shown in the snap attached.

in snap one multipliers (2.3,1.2,1.3,2.6 and 1.13) are randomly generated. likewise for snap two and snap three.

whats the most efficient way to perform this operation as its a simplified version of the original problem (which has over 30k rows). Could use loop but its going to be very, very inefficient.

Answer 1

You want the value of each row to be the product of subsequent rows with random values (random values recalculated for each operation). You can do that as follows:

values = df.sort_index(ascending=False)['Tval']
values = values.expanding().apply(lambda x: np.sum(x*np.random.random(size=len(x))))
df["values"] = values

result:

   Year  Tval     values
0  2021     1  10.342499
1  2022     9  15.595990
2  2023     8  11.491088
3  2024     1   5.447966
4  2025     6   3.689064

Explanation:

• reverse the row order so expanding operates on all rows for first index, one row for last
• apply expanding() to sum rows of greater or equal index, randomly weighted. Weights are recalculated each iteration.
• adds "values" to original dataframe (assignment/join is done on index value, no need to sort the series before adding to df)

As a sanity check, remove the random weighting and observe that this reduces to a reverse cumsum operation:

values = df.sort_index(ascending=False)['Tval']
values = values.expanding().apply(sum)
df["values"] = values

A similar solution can be used if the weights need not change between iterations. As one of the other solutions suggest, you could also pre-calculate all the random weights and take an inner product. This will be memory inefficient but may be significantly faster, as apply is not vectorized.

Answer 2

The operation you're performing is a dot product, where you can account for the decremental use of data by setting weights to 0

weights = np.random.rand(5, 5)
weights = np.tril(weights)

print(weights)
[[0.80446016 0.         0.         0.         0.        ]
 [0.38560755 0.45014049 0.         0.         0.        ]
 [0.61068876 0.91918189 0.66418596 0.         0.        ]
 [0.78442001 0.63551564 0.35635216 0.14712083 0.        ]
 [0.54315584 0.20083916 0.28262627 0.01919842 0.58714358]]

The dot product will the first row of weights , multiply it by the values of df["Tval"] , then sum each of those products. Then it will take the 2nd row of weights and do the same thing, however since we set the first value in the 2nd row of weights to 0, we will essentially ignore the first value of df["Tval"] and multiply/sum the rest of the values. So on and so forth.

df["value"] = df["Tval"] @ weights
print(df)
   Year  Tval      value
0  2021     1  19.181775
1  2022     9  11.324420
2  2023     8   7.936429
3  2024     1   5.792162
4  2025     6   5.243747