Plotting 1D Time Series from 2D Hourly DataFrame

Question

I'm trying to plot a year's worth of utility data downloaded from my utility provider. The data is provided in a matrix where each row is a different day (most recent at the top), and each column is an hour of the day (11:00 AM, 12:00 PM, 1:00 PM, etc). I'd like to transform this 2D DataFrame into a 1D Timeseries, then plot the series.

Using .stack() gets me close, but I can't seem to create a datetime from the date and time column after they are stacked. Also, when plotted it plots the hours correctly from left to right, but the dates descend from left to right. For example it plots the 25th (1am, 2am 3am, etc), 24th (1am, 2am, 3am, etc), 23rd (1am, 2am, 3am, etc). I'm sure this will fix itself after a true datetime is created.

The code below generates a small sample df, but in the real data set all 24 hours are columns and all dates of the year are rows.

df=pd.DataFrame({'Date':['06/25/17','06/24/17','06/23/17'], '12:00 AM':
[1,2,3],'1:00 AM':[4,5,6],'2:00 AM':[7,8,9],})
df.set_index(['Date'], inplace = True)
df

The goal would be to have a series where the index is the time series and the utility usage is the data.

Thank you!

Answer 1

I think you need to unstack your data frame, concatenate the columns Date and level_0 to make a time stamp. Then set the index to the timestamp and drop the extra columns.

df=pd.DataFrame({'Date':['06/25/17','06/24/17','06/23/17'], '12:00 AM':
[1,2,3],'1:00 AM':[4,5,6],'2:00 AM':[7,8,9],})
df.set_index(['Date'], inplace = True)

#Unstack and reset index
df = df.unstack().reset_index()

#concatenate timestamp and convert to datetime
df['Timestamp'] = df['Date'] + ' '+ df['level_0']
df['Timestamp'] = pd.to_datetime(df['Timestamp'],format="%m/%d/%y %I:%M %p")
df =df.sort_values(by='Timestamp')
df = df.set_index('Timestamp')

#drop extra columns
df = df.drop(['Date','level_0'],axis=1)

returns df looking like:

                     0
Timestamp             
2017-06-23 00:00:00  3
2017-06-23 01:00:00  6
2017-06-23 02:00:00  9
2017-06-24 00:00:00  2
2017-06-24 01:00:00  5
2017-06-24 02:00:00  8
2017-06-25 00:00:00  1
2017-06-25 01:00:00  4
2017-06-25 02:00:00  7

You could then plot your time series with

df.plot()

Yielding: