Issue with Pandas resampling

Question

I have a list of dictionaries as follows:

>>>L=[
   {
   "timeline": "2014-10", 
   "total_prescriptions": 17
   }, 
   {
   "timeline": "2014-11", 
   "total_prescriptions": 14
   }, 
   {
   "timeline": "2014-12", 
   "total_prescriptions": 8
  },
  {
  "timeline": "2015-1", 
  "total_prescriptions": 4
  }, 
  {
  "timeline": "2015-3", 
  "total_prescriptions": 10
  }, 
  {
  "timeline": "2015-4", 
  "total_prescriptions": 3
  } 
  ]

What I need to do is to fill missing months,in this case Feb 2015 with total prescription as zero.I used Pandas for it as follows:

>>> df = pd.DataFrame(L)
>>> df.index=pd.to_datetime(df.timeline,format='%Y-%m')
>>> df
           timeline  total_prescriptions
timeline
2014-10-01  2014-10                  17
2014-11-01  2014-11                  14
2014-12-01  2014-12                   8
2015-01-01  2015-1                    4
2015-03-01  2015-3                   10
2015-04-01  2015-4                    3

>>> df = df.resample('MS').fillna(0)
>>> df
            total_prescriptions
timeline
2014-10-01                   17
2014-11-01                   14
2014-12-01                    8
2015-01-01                    4
2015-02-01                    0
2015-03-01                   10
2015-04-01                    3

So far so good..Just what I want..Now i need to convert this data frame back to a list of dicts..This is how I do it :

>>> response = df.T.to_dict().values()
>>> response
[{'total_prescriptions': 0.0}, 
 {'total_prescriptions': 17.0},     
 {'total_prescriptions': 10.0}, 
 {'total_prescriptions': 14.0}, 
 {'total_prescriptions': 4.0}, 
 {'total_prescriptions': 8.0}, 
 {'total_prescriptions': 3.0}]

The ordering is lost,the timeline is missing and total_prescriptions becomes a decimal value from int.What is going wrong ?

Answer 1

Firstly the conversion to decimal is really float dtype due to the resampling as this will introduce NaN values for missing values, you can fix this using astype , you can then restore your 'timeline' column which get lost as it can't figure out how to resample a str so we can apply strftime to the index:

In [80]:
df = df.resample('MS').fillna(0).astype(np.int32)
df['timeline'] = df.index.to_series().apply(lambda x: dt.datetime.strftime(x, '%Y-%m'))
df

Out[80]:
            total_prescriptions timeline
timeline                                
2014-10-01                   17  2014-10
2014-11-01                   14  2014-11
2014-12-01                    8  2014-12
2015-01-01                    4  2015-01
2015-02-01                    0  2015-02
2015-03-01                   10  2015-03
2015-04-01                    3  2015-04

Now we need to sort the dict keys as calling values will lost the sorted order and we can perform a list comprehension to get back to the original form:

In [84]:
d = df.T.to_dict()
[d[key[0]] for key in sorted(d.items())]

Out[84]:
[{'timeline': '2014-10', 'total_prescriptions': 17},
 {'timeline': '2014-11', 'total_prescriptions': 14},
 {'timeline': '2014-12', 'total_prescriptions': 8},
 {'timeline': '2015-01', 'total_prescriptions': 4},
 {'timeline': '2015-02', 'total_prescriptions': 0},
 {'timeline': '2015-03', 'total_prescriptions': 10},
 {'timeline': '2015-04', 'total_prescriptions': 3}]