Pandas 同时为每组填充缺失的日期和值

Question

I have a dataframe (mydf) with dates for each group in monthly frequency like below:

Dt          Id  Sales
2021-03-01  B   2
2021-04-01  B   42
2021-05-01  B   20
2021-06-01  B   4
2020-10-01  A   47
2020-11-01  A   67
2020-12-01  A   46

I want to fill the dt for each group till the Maximum date within the date column starting from the date of Id while simultaneously filling in 0 for the Sales column. So each group starts at their own start date but ends at the same end date.

So for eg ID=A will start from 2020-10-01 and go all the way to 2021-06-03 and the value for the filled dates will be 0.

So the output will be

Dt          Id  Sales
2021-03-01  B   2
2021-04-01  B   42
2021-05-01  B   20
2021-06-01  B   4
2020-10-01  A   46
2020-11-01  A   47
2020-12-01  A   67
2021-01-01  A   0
2021-02-01  A   0
2021-03-01  A   0
2021-04-01  A   0
2021-05-01  A   0
2021-06-01  A   0

I have tried reindex but instead of adding daterange manually I want to use the dates in the groups.

My code is :

f = lambda x: x.reindex(pd.date_range('2020-10-01', '2021-06-01', freq='MS', name='Dt'))
mydf = mydf.set_index('Dt').groupby('Id').apply(f).drop('Id', axis=1).fillna(0)
mydf = mydf.reset_index()

Answer 1

An alternative using pd.MultiIndex with list comprehension:

s = (pd.MultiIndex.from_tuples([[x, d]
      for x, y in df.groupby("Id")["Dt"]
      for d in pd.date_range(min(y), max(df["Dt"]), freq="MS")], names=["Id", "Dt"]))

print (df.set_index(["Id", "Dt"]).reindex(s, fill_value=0).reset_index())

Answer 2

Let's try:

1. Getting the minimum value per group using groupby.min
2. Add a new column to the aggregated mins called max which stores the maximum values from the frame using Series.max on Dt
3. Create individual date_range per group based on the min and max values
4. Series.explode into rows to have a DataFrame that represents the new index.
5. Create a MultiIndex.from_frame to reindex the DataFrame with.
6. reindex with midx and set the fillvalue=0

# Get Min Per Group
dates = mydf.groupby('Id')['Dt'].min().to_frame(name='min')
# Get max from Frame
dates['max'] = mydf['Dt'].max()

# Create MultiIndex with separate Date ranges per Group
midx = pd.MultiIndex.from_frame(
    dates.apply(
        lambda x: pd.date_range(x['min'], x['max'], freq='MS'), axis=1
    ).explode().reset_index(name='Dt')[['Dt', 'Id']]
)

# Reindex
mydf = (
    mydf.set_index(['Dt', 'Id'])
        .reindex(midx, fill_value=0)
        .reset_index()
)

mydf :

           Dt Id  Sales
0  2020-10-01  A     47
1  2020-11-01  A     67
2  2020-12-01  A     46
3  2021-01-01  A      0
4  2021-02-01  A      0
5  2021-03-01  A      0
6  2021-04-01  A      0
7  2021-05-01  A      0
8  2021-06-01  A      0
9  2021-03-01  B      2
10 2021-04-01  B     42
11 2021-05-01  B     20
12 2021-06-01  B      4

---

DataFrame:

import pandas as pd

mydf = pd.DataFrame({
    'Dt': ['2021-03-01', '2021-04-01', '2021-05-01', '2021-06-01', '2020-10-01',
           '2020-11-01', '2020-12-01'],
    'Id': ['B', 'B', 'B', 'B', 'A', 'A', 'A'],
    'Sales': [2, 42, 20, 4, 47, 67, 46]
})
mydf['Dt'] = pd.to_datetime(mydf['Dt'])

Answer 3

Here is a different approach:

from itertools import product

# compute the min-max date range
date_range = pd.date_range(*mydf['Dt'].agg(['min', 'max']), freq='MS', name='Dt')

# make MultiIndex per group, keep only values above min date per group
idx = pd.MultiIndex.from_tuples([e for Id,Dt_min in mydf.groupby('Id')['Dt'].min().items()
                                   for e in list(product(date_range[date_range>Dt_min],
                                                         [Id]))
                                ])

# concatenate the original dataframe and the missing indexes
mydf = mydf.set_index(['Dt', 'Id'])
mydf = pd.concat([mydf,
                  mydf.reindex(idx.difference(mydf.index)).fillna(0)]
                ).sort_index(level=1).reset_index()

mydf

output:

           Dt Id  Sales
0  2020-10-01  A   47.0
1  2020-11-01  A   67.0
2  2020-12-01  A   46.0
3  2021-01-01  A    0.0
4  2021-02-01  A    0.0
5  2021-03-01  A    0.0
6  2021-04-01  A    0.0
7  2021-05-01  A    0.0
8  2021-06-01  A    0.0
9  2021-03-01  B    2.0
10 2021-04-01  B   42.0
11 2021-05-01  B   20.0
12 2021-06-01  B    4.0

Answer 4

We can use the complete function from pyjanitor to expose the missing values:

Convert Dt to datetime:

 df['Dt'] = pd.to_datetime(df['Dt'])

Create a mapping of Dt to new values, via pd.date_range , and set the frequency to monthly begin ( MS ):

 max_time = df.Dt.max()

 new_values = {"Dt": lambda df:pd.date_range(df.min(), max_time, freq='1MS')}

# pip install pyjanitor
import janitor
import pandas as pd
df.complete([new_values], by='Id').fillna(0)


   Id         Dt  Sales
0   A 2020-10-01   47.0
1   A 2020-11-01   67.0
2   A 2020-12-01   46.0
3   A 2021-01-01    0.0
4   A 2021-02-01    0.0
5   A 2021-03-01    0.0
6   A 2021-04-01    0.0
7   A 2021-05-01    0.0
8   A 2021-06-01    0.0
9   B 2021-03-01    2.0
10  B 2021-04-01   42.0
11  B 2021-05-01   20.0
12  B 2021-06-01    4.0

Sticking to Pandas only, we can combine apply , with groupby and reindex ; thankfully, Dt is unique, so we can safely reindex:

(df
 .set_index('Dt')
 .groupby('Id')
 .apply(lambda df: df.reindex(pd.date_range(df.index.min(), 
                                            max_time, 
                                            freq='1MS'), 
                              fill_value = 0)
                              )
 .drop(columns='Id')
 .rename_axis(['Id', 'Dt'])
 .reset_index())
 
   Id         Dt  Sales
0   A 2020-10-01     47
1   A 2020-11-01     67
2   A 2020-12-01     46
3   A 2021-01-01      0
4   A 2021-02-01      0
5   A 2021-03-01      0
6   A 2021-04-01      0
7   A 2021-05-01      0
8   A 2021-06-01      0
9   B 2021-03-01      2
10  B 2021-04-01     42
11  B 2021-05-01     20
12  B 2021-06-01      4