Pandas DataFrame: Complex linear interpolation

Question

I have a dataframe with 4 sections

Section 1: Product details

Section 2: 6 Potential product values based on a range of simulations

Section 3: Upper and lower bound for the input parameter to the simulations

Section 4: Randomly generated values for the input parameters

---

Section 2 is generated by pricing the product at equal intervals between the upper and lower bound.

I need to take the values in Section 4 and figure out the corresponding product value. Here is a possible setup for this dataframe:

table2 = pd.DataFrame({
        'Product Type': ['A', 'B', 'C', 'D'],
        'State_1_Value': [10, 11, 12, 13],
    'State_2_Value': [20, 21, 22, 23],
    'State_3_Value': [30, 31, 32, 33],
    'State_4_Value': [40, 41, 42, 43],
    'State_5_Value': [50, 51, 52, 53],
    'State_6_Value': [60, 61, 62, 63],
    'Lower_Bound': [-1, 1, .5, 5],
    'Upper_Bound': [1, 2, .625, 15],
    'sim_1': [0, 0, .61, 7],
    'sim_2': [1, 1.5, .7, 9],
    })

>>> table2
   Lower_Bound Product Type  State_1_Value  State_2_Value  State_3_Value  \
0         -1.0            A             10             20             30   
1          1.0            B             11             21             31   
2          0.5            C             12             22             32   
3          5.0            D             13             23             33   

   State_4_Value  State_5_Value  State_6_Value  Upper_Bound  sim_1  sim_2  
0             40             50             60        1.000    0.0    1.0  
1             41             51             61        2.000    0.0    1.5  
2             42             52             62        0.625    0.61    0.7  
3             43             53             63       15.000    7.0    9.0  

I will run through a couple examples of this calculation to make it clear what my question is.

Product A - sim_2 The input here is 1.0. This is equal to the upper bound for this product. Therefore the simulation value is equivalent to the state_6 value - 60

Product B - sim_2 The input here is 1.5. the LB to UB range is (1,2), therefore the 6 states are {1,1.2,1.4,1.6,1.8,2}. 1.5 is exactly in the middle of state_3 which has a value of 31 and state 4 which has a value of 41. Therefore the simulation value is 36.

Product C - sim_1 The input here is .61. The LB to UB range is (.5,.625), therefore the 6 states are {.5,.525,.55,.575,.6,.625}. .61 is between state 5 and 6. Specifically the bucket it would fall under would be 5*(.61-.5)/(.625-.5)+1 = 5.4 (it is multiplied by 5 as that is the number of intervals - you can calculate it other ways and get the same result). Then to calculate the value we use that bucket in a weighing of the values for state 5 and state 6: (62-52)*(5.4-5)+52 = 56.

Product B - sim_1 The input here is 0 which is below the lower bound of 1. Therefore we need to extrapolate the value. We use the same formula as above we just use the values of state 1 and state 2 to extrapolate. The bucket would be 5*(0-1)/(2-1)+1 = -4. The two values used at 11 and 21, so the value is (21-11)*(-4-1)+11= -39

I've also simplified the problem to try to visualize the solution, my final code needs to run on 500 values and 10,000 simulations, and the dataframe will have about 200 rows.

Here are the formulas I've used for the interpolation although I'm not committed to them specifically.

Bucket = N*(sim_value-LB)/(UB-LB) + 1 where N is the number of intervals

then nLower is the state value directly below the bucket, and nHigher is the state value directly above the bucket. If the bucket is outside the UB/LB, then force nLower and nHigher to be either the first two or last two values.

Final_value = (nHigher-nLower)*(Bucket1 - number_value_of_nLower)+nLower

To summarize, my question is how I can generate the final results based on the combination of input data provided. The most challenging part to me is how to make the connection from the Bucket number to the nLower and nHigher values.

Answer 1

I was able to generate the result using the following code. I'm not sure of the memory implications on a large dataframe, so still interested in better answers or improvements.

Edit: Ran this code on the full dataset, 141 rows, 500 intervals, 10,000 simulations, and it took slightly over 1.5 hours. So not quite as useless as I assumed, but there is probably a smarter way of doing this in a tiny fraction of that time.

for i in range(1,3):
    table2['Bucket%s'%i] = 5 * (table2['sim_%s'%i] - table2['Lower_Bound']) / (table2['Upper_Bound'] - table2['Lower_Bound']) + 1
    table2['lv'] = table2['Bucket%s'%i].map(int)
    table2['hv'] = table2['Bucket%s'%i].map(int) + 1
    table2.ix[table2['lv'] < 1 , 'lv'] = 1
    table2.ix[table2['lv'] > 5 , 'lv'] = 5
    table2.ix[table2['hv'] > 6 , 'hv'] = 6
    table2.ix[table2['hv'] < 2 , 'hv'] = 2
    table2['nLower'] = table2.apply(lambda row: row['State_%s_Value'%row['lv']],axis=1)
    table2['nHigher'] = table2.apply(lambda row: row['State_%s_Value'%row['hv']],axis=1)
    table2['Final_value_%s'%i] = (table2['nHigher'] - table2['nLower'])*(table2['Bucket%s'%i]-table2['lv']) + table2['nLower']

Output:

>>> table2
   Lower_Bound Product Type  State_1_Value  State_2_Value  State_3_Value  \
0         -1.0            A             10             20             30   
1          1.0            B             11             21             31   
2          0.5            C             12             22             32   
3          5.0            D             13             23             33   

   State_4_Value  State_5_Value  State_6_Value  Upper_Bound  sim_1  sim_2  \
0             40             50             60        1.000   0.00    1.0   
1             41             51             61        2.000   0.00    1.5   
2             42             52             62        0.625   0.61    0.7   
3             43             53             63       15.000   7.00    9.0   

   Bucket1  lv  hv  nLower  nHigher  Final_value_1  Bucket2  Final_value_2  
0      3.5   5   6      50       60           35.0      6.0           60.0  
1     -4.0   3   4      31       41          -39.0      3.5           36.0  
2      5.4   5   6      52       62           56.0      9.0           92.0  
3      2.0   3   4      33       43           23.0      3.0           33.0 

Answer 2

I posted a superior solution with no loops here:

Alternate method to avoid loop in pandas dataframe

df= pd.DataFrame({
            'Product Type': ['A', 'B', 'C', 'D'],
            'State_1_Value': [10, 11, 12, 13],
        'State_2_Value': [20, 21, 22, 23],
        'State_3_Value': [30, 31, 32, 33],
        'State_4_Value': [40, 41, 42, 43],
        'State_5_Value': [50, 51, 52, 53],
        'State_6_Value': [60, 61, 62, 63],
        'Lower_Bound': [-1, 1, .5, 5],
        'Upper_Bound': [1, 2, .625, 15],
        'sim_1': [0, 0, .61, 7],
        'sim_2': [1, 1.5, .7, 9],
        })


buckets = df.ix[:,-2:].sub(df['Lower_Bound'],axis=0).div(df['Upper_Bound'].sub(df['Lower_Bound'],axis=0),axis=0) * 5 + 1
low = buckets.applymap(int)
high = buckets.applymap(int) + 1
low = low.applymap(lambda x: 1 if x < 1 else x)
low = low.applymap(lambda x: 5 if x > 5 else x)
high = high.applymap(lambda x: 6 if x > 6 else x)
high = high.applymap(lambda x: 2 if x < 2 else x)
low_value = pd.DataFrame(df.filter(regex="State|Type").values[np.arange(low.shape[0])[:,None], low])
high_value = pd.DataFrame(df.filter(regex="State|Type").values[np.arange(high.shape[0])[:,None], high])
df1 = (high_value - low_value).mul((buckets - low).values) + low_value
df1['Product Type'] = df['Product Type']