Extend datetimeindex to previous times in pandas

Question

MRE:

idx = pd.date_range('2015-07-03 08:00:00', periods=30, 

freq='H')
data = np.random.randint(1, 100, size=len(idx))
df = pd.DataFrame({'index':idx, 'col':data})

df.set_index("index", inplace=True)

which looks like:

                    col
index   
2015-07-03 08:00:00 96
2015-07-03 09:00:00 79
2015-07-03 10:00:00 15
2015-07-03 11:00:00 2
2015-07-03 12:00:00 84
2015-07-03 13:00:00 86
2015-07-03 14:00:00 5
.
.
.

Note that dataframe contain multiple days. Since frequency is in hours, starting from 07/03 08:00:00 it will contain hourly date.

I want to get all data from 05:00:00 including day 07/03 even if it will contain value 0 in "col" column.

I want to extend it backwards so it starts from 05:00:00.

No I just can't start from 05:00:00 since I already have dataframe that starts from 08:00:00. I am trying to keep everything same but add 3 rows in the beginning to include 05:00:00, 06:00:00, and 07:00:00

Answer 1

The reindex method is handy for changing the index values:

idx = pd.date_range('2015-07-03 08:00:00', periods=30, freq='H')
data = np.random.randint(1, 100, size=len(idx))

# use the index param to set index or you might lose the freq 
df = pd.DataFrame({'col':data}, index=idx)

# reindex with a new index
start = df.tshift(-3).index[0]
end = df.index[-1]
new_index = pd.date_range(start, end, freq='H')
new_df = df.reindex(new_index)

resample is also very useful for date indices

Answer 2

Just change the time from 08:00:00 to 05:00:00 in your code and create 3 more rows and update this dataframe to the existing one.

idx1 = pd.date_range('2015-07-03 05:00:00', periods=3,freq='H')

df1 = pd.DataFrame({'index': idx1 ,'col':np.random.randint(1,100,size = 3)})
df1.set_index('index',inplace=True)
df = df1.append(df)
print(df)

Add this snippet to your code...