pandas 中的条件加权平均计算

Question

I have 2 Dataframes as below

Teacher_Commission_df as below

+---------+---------+----------+---------+
| Subject |  Harare | Redcliff |  Norton |
+---------+---------+----------+---------+
| Science |  0.100  |   0.125  |  0.145  |
+---------+---------+----------+---------+
| English |  0.125  |   0.150  |  0.170  |
+---------+---------+----------+---------+
|  Maths  |  0.090  |   0.115  |  0.135  |
+---------+---------+----------+---------+
|  Music  |  0.100  |   0.125  |  0.145  |
+---------+---------+----------+---------+
|  Total  |  0.415  |   0.515  |  0.595  |
+---------+---------+----------+---------+

Students_df as below. (Note No students for Maths in Harare and Norton )

+---------+--------+----------+--------+
| Subject | Harare | Redcliff | Norton |
+---------+--------+----------+--------+
| Science |   15   |    18    |   20   |
+---------+--------+----------+--------+
| English |   35   |    33    |   31   |
+---------+--------+----------+--------+
|  Maths  |        |    25    |        |
+---------+--------+----------+--------+
|  Music  |   40   |    42    |   45   |
+---------+--------+----------+--------+

I need to calculate the weighted average commission of each city, with a condition.

First of all I'll give the desired output and explain the methodology.

desired output is s below.

+------------+--------+----------+--------+
| Total_Paid | Harare | Redcliff | Norton |
+------------+--------+----------+--------+
|   Science  |  4.62  |   4.37   |  6.30  |
+------------+--------+----------+--------+
|   English  |  13.46 |   9.61   |  11.46 |
+------------+--------+----------+--------+
|    Maths   |  0.00  |   5.58   |  0.00  |
+------------+--------+----------+--------+
|    Music   |  12.31 |   10.19  |  14.18 |
+------------+--------+----------+--------+

Calculation methodology

if in any city column [Harare, Redcliff, Norton] , if students of any Subject [Science, English, Maths, Music] is zero then that particular subject 's Teacher_Commission should be removed in the weight.

For example, in Students_df : Take city Harare column of Science subject. since, Maths is zero in Harare , the teacher_Commission will be calculated as follows. 15 * [0.10 / (0.415 - 0.09)] = 4.62 note the 0.09 removal in the denominator of the total. where as in Radcliff it is calculated without removal as 18 * [0.125 / 0.515] = 4.37

I hope my explanation is clear.

This can be easily done in Microsoft Excel by using an IF condition. But, I'm looking for a scalable pandas solution.

I'm not sure how to start the calculation process. Hence, please give me a kick start to solve this.

-----------------------------------------------------------------------------------------
 UPDATE
  I've managed to solve this. Refer to my answer below and suggest for any improvements
------------------------------------------------------------------------------------------

Answer 1

So, what you need is the row/column index of every empty-null value in the dataframe?

You can use numpy.where(). Depending on the data type of your null object you could

1. Load df as np array
2. I,j = np.where(“NaN”)
3. i and j are now indexes you can use to eliminate the weights if the sizes are same or use dataframe.index to find which weight to remove.

Replace NaN with Null or “” depending on your dtype

This is similar to what you'd do in excel using an IF

Personally I would just make a copy dataframe binary ie put a 1 wherever there is a non null value in the dataframe and 0 at null location, then just miltiple the two vectors. But thats probably more processing overhead

Answer 2

Solution using pandas

This is actually just two lines of code using pandas:

import numpy as np
df_tmp = teacher_commission_df[~students_df.isnull()]
df = (df_tmp.div(df_tmp.apply(np.nansum, axis=0)) * students_df).fillna(0)

Outcome ^{(With the new 3 digits precision data.)}

In [1]: df
Out[1]:
            Harare   Redcliff     Norton
Subject
Science   4.615385   4.368932   6.304348
English  13.461538   9.611650  11.456522
Maths     0.000000   5.582524   0.000000
Music    12.307692  10.194175  14.184783

Explanation of the code above

^{Note : This explanation uses the 2 digit precision data given in the original question.}

• First, you may use boolean indexing, by using the DataFrame.isnull()

In [1]: students_df.isnull()
Out[1]:
         Harare  Redcliff  Norton
Subject
Science   False     False   False
English   False     False   False
Maths      True     False    True
Music     False     False   False

• Then, you can select the non-null values from the teacher_commission_df using boolean indexing and the not operator ( ~ ).

In [3]: teacher_commission_df[~students_df.isnull()]
Out[3]:
         Harare  Redcliff  Norton
Subject
Science    0.10      0.13    0.15
English    0.13      0.15    0.17
Maths       NaN      0.12     NaN
Music      0.10      0.13    0.15

• Let's save this temporary dataframe into new variable, df_tmp :

In [12]: df_tmp = teacher_commission_df[~students_df.isnull()]

• Now, we want to divide value in each cell by the sum of the column values. The sum of column values is calculated, ignoring nans, with the help of apply() and np.nansum :

In [14]: df_tmp.apply(np.nansum, axis=0)
Out[14]:
Harare      0.33
Redcliff    0.53
Norton      0.47
dtype: float64

• Then, combine the summing with division, using DataFrame.div() :

In [15]: df_tmp.div(df_tmp.apply(np.nansum, axis=0))
Out[15]:
           Harare  Redcliff    Norton
Subject
Science  0.303030  0.245283  0.319149
English  0.393939  0.283019  0.361702
Maths         NaN  0.226415       NaN
Music    0.303030  0.245283  0.319149

• Then, multiply the dataframes (elementwise multiplication):

In [16]: df_tmp.div(df_tmp.apply(np.nansum, axis=0)) * students_df
Out[16]:
            Harare   Redcliff     Norton
Subject
Science   4.545455   4.415094   6.382979
English  13.787879   9.339623  11.212766
Maths          NaN   5.660377        NaN
Music    12.121212  10.301887  14.361702

• Lastly, fill NaN values with zeroes with DataFrame.fillna() :

In [17]: (df_tmp.div(df_tmp.apply(np.nansum, axis=0)) * students_df).fillna(0)
Out[17]:
            Harare   Redcliff     Norton
Subject
Science   4.545455   4.415094   6.382979
English  13.787879   9.339623  11.212766
Maths     0.000000   5.660377   0.000000
Music    12.121212  10.301887  14.361702

Answer 3

Based on the suggestion given by User: aak . I've managed to solve this purely from numpy .

# Load data and fill N/A values
Teacher_Commission_df = pd.read_excel('data_Teacher.xlsx',index_col='Subject', skipfooter=1)
Students_df = pd.read_excel('data_Studenst.xlsx',index_col='Subject')
Students_df.fillna(value=0, inplace= True)


# Convert Dataframes to Numpy Arrays
T = Teacher_Commission_df.to_numpy(dtype='float')
S = Students_df.to_numpy(dtype='float')

# Filter index of ZERO values from Students Numpy Array and 
# replace the correponding Values in teachers Numpy Array
T[np.where(S == 0)] = 0

# creat a temporary Sum numpy array for calculation
Total_Teacher = T.sum(axis=0)

#calculate incentives
Calculations = T * (S/Total_Teacher)

incentives = (pd.DataFrame(Calculations, columns=Students_df.columns, index=Students_df.index)
                  .round(decimals=2)
                  .reset_index())
incentives