我可以 pivot 基于元组列表的 pandas dataframe 中的一列吗？

Question

I am trying to change the structure of my dataframe as follows. I have a dataset with historical KPI information: each record contains a date, the KPI ID, multiple dimensions and the KPI value.

Based on a list of 3-tuples, I want to transform this dataframe such that the end result is a combination of 2 records in the existing dataframe with a numerator and a denominator, each coming from an individual record with the same date/dimensions.

Current dataframe:

Date  | KPI_ID | Dimension | Value
Apr 5 | KPI_1  | Lorem     | 1
Apr 5 | KPI_2  | Lorem     | 3
Apr 5 | KPI_1  | Ipsum     | 4
Apr 5 | KPI_2  | Ipsum     | 8
Apr 5 | KPI_3  | Dolor     | 2
Apr 5 | KPI_4  | Dolor     | 2

List of 3-tuples giving the combinations of KPI_IDs such as [Result_ID, KPI_Numerator, KPI_Denominator]:

[['Result_1', 'KPI_1', 'KPI_2'], ['Result_2', 'KPI_3', 'KPI_4']]

Desired result:

Date  | Result_ID | Dimension | Numerator | Denominator
Apr 5 | Result_1  | Lorem     | 1         | 3
Apr 5 | Result_1  | Ipsum     | 4         | 8
Apr 5 | Result_2  | Dolor     | 2         | 2

I have tried to use df.merge and df.groupby with an aggregation function and but am struggling to understand how I can best integrate the list of tuples in the equation. Looping over the dataframe does not seem to be the answer because I would have to manually look for the record with exactly the same dimensions which I do not think is performant.

Answer 1

You can create a dataframe with combinations (Result_ID, KPI_1, KPI_2), then merge it twice to the original dataframe based on KPI_1 first, and then on KPI_2 (this time also matching on Time and Dimension):

# Create combinations dataframe
cs = [['Result_1', 'KPI_1', 'KPI_2'], ['Result_2', 'KPI_3', 'KPI_4']]
df_cs = pd.DataFrame(cs, columns=['Result_ID', 'KPI_1', 'KPI_2'])

# Merge combinations dataframe to original data:
# 1. So that 'KPI_1' in combinations = 'KPI_ID' in data
# 2. So that 'KPI_2' in combinations = 'KPI_ID' in data,
#    and we get a match on ['Date', 'Dimension']
cols = ['Date', 'Result_ID', 'Dimension', 'Numerator', 'Denominator']
df_out = (df_cs
    .merge(df.rename(columns={'Value': 'Numerator'}),
           left_on='KPI_1', right_on='KPI_ID')
    .drop(columns='KPI_ID')
    .merge(df.rename(columns={'Value': 'Denominator'}),
           left_on=['Date', 'Dimension', 'KPI_2'],
           right_on=['Date', 'Dimension', 'KPI_ID'])
    .drop(columns=['KPI_ID', 'KPI_1', 'KPI_2'])
)[cols]

Output:

    Date Result_ID Dimension  Numerator  Denominator
0  Apr 5  Result_1     Lorem          1            3
1  Apr 5  Result_1     Ipsum          4            8
2  Apr 5  Result_2     Dolor          2            2

Answer 2

I am trying to change the structure of my dataframe as follows. I have a dataset with historical KPI information: each record contains a date, the KPI ID, multiple dimensions and the KPI value.

Based on a list of 3-tuples, I want to transform this dataframe such that the end result is a combination of 2 records in the existing dataframe with a numerator and a denominator, each coming from an individual record with the same date/dimensions.

Current dataframe:

Date  | KPI_ID | Dimension | Value
Apr 5 | KPI_1  | Lorem     | 1
Apr 5 | KPI_2  | Lorem     | 3
Apr 5 | KPI_1  | Ipsum     | 4
Apr 5 | KPI_2  | Ipsum     | 8
Apr 5 | KPI_3  | Dolor     | 2
Apr 5 | KPI_4  | Dolor     | 2

List of 3-tuples giving the combinations of KPI_IDs such as [Result_ID, KPI_Numerator, KPI_Denominator]:

[['Result_1', 'KPI_1', 'KPI_2'], ['Result_2', 'KPI_3', 'KPI_4']]

Desired result:

Date  | Result_ID | Dimension | Numerator | Denominator
Apr 5 | Result_1  | Lorem     | 1         | 3
Apr 5 | Result_1  | Ipsum     | 4         | 8
Apr 5 | Result_2  | Dolor     | 2         | 2

I have tried to use df.merge and df.groupby with an aggregation function and but am struggling to understand how I can best integrate the list of tuples in the equation. Looping over the dataframe does not seem to be the answer because I would have to manually look for the record with exactly the same dimensions which I do not think is performant.

Answer 3

Let's create a combination within each mapping, pairing Result with the KPIs . Create a dataframe of this combination, merge with the original dataframe, pivot, and conclude with some massaging, to get the data in the final form that OP desires.

The caveat with this is pivot requires unique combination of index and columns ; for the data shared, there is no need to worry about this.

from itertools import product, chain
mapping = [['Result_1', 'KPI_1', 'KPI_2'], ['Result_2', 'KPI_3', 'KPI_4']]
maps = (product([left], [*right]) for left, *right in mapping)
maps = chain.from_iterable(maps)
maps = pd.DataFrame(maps, columns=['Result_ID', 'KPI_ID'])

maps
  Result_ID KPI_ID
0  Result_1  KPI_1
1  Result_1  KPI_2
2  Result_2  KPI_3
3  Result_2  KPI_4

(df
 .merge(maps, how='left', on='KPI_ID')
 .assign(KPI_ID = lambda df: df.KPI_ID.map({"KPI_1":"Numerator",           
                                            "KPI_2":"Denominator",
                                            "KPI_3":"Numerator",
                                            "KPI_4":"Denominator"}),
         sorter = lambda df: df.Dimension.factorize()[0])
 .pivot(['Date','Result_ID','Dimension', 'sorter'],  
         'KPI_ID', 
         'Value')
 .rename_axis(columns=None)
 .sort_values('sorter')
 .droplevel('sorter')
 .iloc[:, ::-1]
 .reset_index()
 )

    Date Result_ID Dimension  Numerator  Denominator
0  Apr 5  Result_1     Lorem          1            3
1  Apr 5  Result_1     Ipsum          4            8
2  Apr 5  Result_2     Dolor          2            2