python iteration over each bins in a dataframe

Question

I'm working on a measurement dataset. I have data in a dataframe like in the below table, the data is power and windspeed for every 0.5m/s windspeed bin. But I need to calculate the Sensitivity column at each bin from power and windspeed values. The formula for sensitivity is

sensitivity = abs ( (Pi - Pi_1) / (Vi - Vi_1) )

We have to subtract the current bin values of power and speed from previous bin value.

I need a for loop script for this scenario. I'm really a bit confused by using all for loop options, can someone help me on this please?

Note: I got these values from the below DataFrame script:

uncut = df.groupby(pd.cut(df.normalized_speed, ws_bin))['pt_power_avg', 'normalized_speed'].mean() 

Data Table:

normalized_speed    pt_power_avg [Pi] normalized_speed [Vi]  *sensitivity*
[Ci]"
(0, 0.5]                 0                        0                   -   
(0.5, 1]                 0                        0                   -   
(1, 1.5]                 0                        0                   -   
(1.5, 2]                 0                        0                   -   
(2, 2.5]                 6.46                     2.44               2.6 
(2.5, 3]                14.22                     2.73              26.2 
(3, 3.5]                27.05                     3.26              24.4 
(3.5, 4]                56.67                     3.77              58.6 
(4, 4.5]                88.55                     4.26              64.7 
(4.5, 5]               121.95                     4.76              66.8 
(5, 5.5]               166.87                     5.26              89.5 
(5.5, 6]               221.16                     5.74             112.6 
(6, 6.5]               283.94                     6.26             122.4 
(6.5, 7]               310.32                     6.74              54.7 
(7, 7.5]               472.59                     7.29             297.0 
(7.5, 8]               582.02                     7.70             261.2 
(8, 8.5]               703.98                     8.17             261.1 
(8.5, 9]               927.60                     8.77             375.4 
(9, 9.5]               995.10                     9.11             194.1    

Answer 1

Use shift() instead of for looping

In pandas, you should use the shift feature instead of for looping . Pandas is built to do these exact type of calculations on columns of data without for looping over each row!

Assuming your original data is in a DataFrame called df , the equation would be represented as

# Calculate equation (broken into numerator and denominator for example)
numerator = df['pt_power_avg [Pi]'] - df['pt_power_avg [Pi]'].shift()
denominator = df['normalized_speed [Vi]'] - df['normalized_speed [Vi]'].shift()
calculated_sensitivity = (numerator / denominator).abs()

# Add to DataFrame
print 'Calculated Sensitivity:'
print calculated_sensitivity
print

---

Calculated Sensitivity:
normalized_speed
(0, 0.5]           NaN
(0.5, 1]           NaN
(1, 1.5]           NaN
(1.5, 2]           NaN
(2, 2.5]      2.647541
(2.5, 3]     26.758621
(3, 3.5]     24.207547
(3.5, 4]     58.078431
(4, 4.5]     65.061224
(4.5, 5]     66.800000
(5, 5.5]     89.840000
(5.5, 6]    113.104167
(6, 6.5]    120.730769
(6.5, 7]     54.958333
(7, 7.5]    295.036364
(7.5, 8]    266.902439
(8, 8.5]    259.489362
(8.5, 9]    372.700000
(9, 9.5]    198.529412
dtype: float64

Answer 2

my result is below without 2.6 in [2,2.5]

       normalized_speed
(0, 0.5]            NaN
(0.5, 1]            NaN
(1, 1.5]            NaN
(1.5, 2]            NaN
(2, 2.5]            NaN
(2.5, 3]      26.180203
(3, 3.5]      24.390952
(3.5, 4]      58.638289
(4, 4.5]      64.677315
(4.5, 5]      66.751720
(5, 5.5]      89.462064
(5.5, 6]     112.621292
(6, 6.5]     122.390346
(6.5, 7]      54.709085
(7, 7.5]     296.962721
(7.5, 8]     261.151143
(8, 8.5]     261.063389
(8.5, 9]     375.387079
(9, 9.5]     194.122176
(9.5, 10]           NaN
dtype: float64