Pandas 矢量化：基于 JSON 文件的累积和

Question

I'm trying to sum a score based on values in a DataFrame and two json -files. I have a minimum example and a minimum solution, but this needs to be vectorized somehow because in the real-case there are over a million rows, and it took ~40min to run through 1% of the rows.

My first.json -file is:

{
    "variables" : {
        "var_1": {
            "values": [
                {
                    "lb": 1.0,
                    "b_cumul": 0.04
                },
                {
                    "lb": 3.0,
                    "b_cumul": 0.28
                }
            ]
        },
        "var_2": {
            "values": [
                {
                    "lb": 0,
                    "b_cumul": -0.09
                },
                {
                    "lb": 1,
                    "b_cumul": 0.14
                },
                {
                    "lb:": 4,
                    "b_cumul": 0.03
                }
            ]
        },
        "var_4": {
            "values": [
                {
                    "lb": "1",
                    "b_cumul": 0.06
                }
            ]
        }
    }
}

My second.json file is:

{
    "variables" : {
        "var_1": {
            "values": [
                {
                    "lb": 1.0,
                    "b_cumul": -0.15
                },
                {
                    "lb": 2.0,
                    "b_cumul": 0.06
                },
                {
                    "lb": 4.0,
                    "b_cumul": 0.02
                },
                {
                    "lb": 5.0,
                    "b_cumul": 0.15
                }
            ]
        },
        "var_3": {
            "values": [
                {
                    "lb": 0.0,
                    "b_cumul": 0.12
                },
                {
                    "lb": 2.0,
                    "b_cumul": 0.25
                }
            ]
        },
        "var_6": {
            "values": [
                {
                    "lb": 0.0,
                    "b_cumul": -0.16
                },
                {
                    "lb": 1.0,
                    "b_cumul": -0.06
                }
            ]
        }
    }
}

This is our initial dataframe:

import pandas as pd
# setup initial test-data
usage = ['first', 'second', 'first', 'second', 'second', 'second', 'first']
var_1 = [-1, -1, 0, 1, 3, 8, 2]
var_2 = [1, 3, -1, 0, 9, 2, 1]
var_3 = [0, 1, 0, -1, -1, 42, 3]
df = pd.DataFrame({'usage': usage, 'var_1': var_1, 'var_2': var_2, 'var_3': var_3})

The variables var_1 , var_2 , var_3 that are available in df , decides which variables we want to look at in the json -files. The scores should be cumulative sums retrieved from the json -files, depending on the values in df .

Looking at my first row, I have (var_1=-1, var_2=1, var_3=0) . Since usage='first' for this same row, I need to check in first.json what scores these variables correspond to. var_3 does not exist in first.json , so this variable gives 0 score. var_1=-1 , so this also gives 0 score. var_2=1 , so here we need to look in first.json and get the scores for values corresponding to <=1 , which in this case is -0.09+0.14=0.05 . So we want to add this information in the dataframe by df.loc[0, 'score_sum']=0+0-0.09+0.14 .

I have solved this with the code below, but as mentioned before this is very inefficient and does not work for a larger df .

import seaborn as sns
from matplotlib.pyplot import plt
import json

# return score based on given value and score ranges
def calculate_score(value: Number, score_ranges: pd.DataFrame) -> Number:
    score_ranges.lb = pd.to_numeric(score_ranges.lb)
    if score_ranges.lb.min() > value:
        return 0
    score_sum = score_ranges.loc[(score_ranges.lb <= value), 'b_cumul'].sum()
    return score_sum

# read relevant data from json files
models = {key: [] for key in df['usage'].unique()}
for path in models:
    f = open(f"{path}.json")
    all_variables = json.load(f)['variables']
    relevant_variables = [x for x in df.columns if x in all_variables]
    for var in relevant_variables:
        models[path].append({var: all_variables[var]['values']})

# calculate scores
df['score_sum'] = np.nan
for index, row in df.iterrows():
    score = 0
    m = row['usage']
    for var in models[m]:
        var_name = list(var)[0]
        value = row[var_name] 
        if value == -1:
            score += 0
        elif value > -1:
            score += calculate_score(value, pd.DataFrame(var[var_name]))
    df.loc[index, 'score_sum'] = score

By running this code and then printing df , we notice that the first row has score_sum=0.05 as intended. We import seaborn, plt at the top because we want to run sns.distplot(df['score_sum']) at the end and save the figure.

EDIT: As requested, see below for a screenshot of the total resulting DataFrame . And just to clarify: For the second row, usage='second' which means we use second.json , however we don't have var_2 in this json so var_2=3 will just add score_sum+=0 , but var_3=1 will add score_sum+=0.12

Answer 1

Solved by looping through the models first as defined by df['usage'].unique() and will have an equivalent amount of json -files. Then we loop through each relevant variable in the json , then each incremental value for the given variable. We create masks inside each loop, and start by the lowest value since we want to cumulatively add the values.

df['new_scores'] = 0
for model in models:
    for var in models[model]:
        var_name = list(var)[0]
        json_values = pd.DataFrame(var[var_name])
        json_values = json_values.set_index('lb')
        for value in json_values.index:
            mask = (df[var_name] >= float(value)) & (df['usage'] == model)
            df.loc[mask, 'new_scores'] += json_values.loc[value, 'b_cumul']

When comparing the column from the old solution with the column from the new solution, the values are exactly the same. For the original dataframe of ~6M rows, it went from taking ~4h to iterate through 5% of the dataframe , to ~70s to run through the whole script.