如何使用增量日期时间模拟 pandas dataframe 数据

Question

In python3 and pandas:

assuming i have a dataframe:

datetime,id,value
2020-03-12,1,100
2020-03-13,1,105
2020-03-14,1,110
2020-03-12,2,100
2020-03-13,2,105
2020-03-14,2,110

I am trying to simulate these datasets with x extra historical days.

Let us say x=2 for now, and we wont add any new ID. Just existing IDs in the datasets. The value column can be incremental or random. Wonder how could I do it?

The first thing we have to is to extend the time:

df2=pd.DataFrame(pd.date_range(pd.to_datetime('today'), periods=10, freq='1440min'))

df['datetime']=df['datetime'].append(df2)

then i got

ValueError: cannot reindex from a duplicate axis

Wonder how could I do it?

Answer 1

one way could be to set_index the datetime and id columns, then reindex with all the dates you want generated through date_range using pd.MultiIndex.from_product and finally reset_index to put them back as columns like:

#ensure datetime is good format
df['datetime'] = pd.to_datetime(df['datetime'])

#set parameter for extra days
x=2
df_re = df.set_index(['id', 'datetime'])\
          .reindex(pd.MultiIndex.from_product([df['id'].unique(), 
                                               pd.date_range(df['datetime'].min(), 
                                                             df['datetime'].max() + pd.Timedelta(days=x))], 
                                              names=['id', 'datetime']),
                   fill_value=120)\
          .reset_index()

print (df_re)
   id   datetime  value
0   1 2020-03-12    100
1   1 2020-03-13    105
2   1 2020-03-14    110
3   1 2020-03-15    120
4   1 2020-03-16    120
5   2 2020-03-12    100
6   2 2020-03-13    105
7   2 2020-03-14    110
8   2 2020-03-15    120
9   2 2020-03-16    120