Use pandas to group list items by nth week

Question

I have some data for next 10 days.

[{'cover_image': 'TODO - s3 link', 'epoch': 1497403800000},
 {'cover_image': 'TODO - s3 link', 'epoch': 1497490200000},
 {'cover_image': 'TODO - s3 link', 'epoch': 1497576600000},
 {'cover_image': 'TODO - s3 link', 'epoch': 1497663000000},
 {'cover_image': 'TODO - s3 link', 'epoch': 1497749400000},
 {'cover_image': 'TODO - s3 link', 'epoch': 1497835800000},
 {'cover_image': 'TODO - s3 link', 'epoch': 1497922200000},
 {'cover_image': 'TODO - s3 link', 'epoch': 1498008600000},
 {'cover_image': 'TODO - s3 link', 'epoch': 1498095000000},
 {'cover_image': 'TODO - s3 link', 'epoch': 1498181400000}]

Using the week number, I want to group the data into this week and next week .

I want something like,

{
    '24': [# list of items for this week],
    '25': [# list of items for next week]
}
# i.e.
{'24': [{'cover_image': 'TODO - s3 link', 'epoch': 1497403800000},
  {'cover_image': 'TODO - s3 link', 'epoch': 1497490200000},
  {'cover_image': 'TODO - s3 link', 'epoch': 1497576600000},
  {'cover_image': 'TODO - s3 link', 'epoch': 1497663000000},
  {'cover_image': 'TODO - s3 link', 'epoch': 1497749400000}],
'25': [{'cover_image': 'TODO - s3 link', 'epoch': 1497835800000},
  {'cover_image': 'TODO - s3 link', 'epoch': 1497922200000},
  {'cover_image': 'TODO - s3 link', 'epoch': 1498008600000},
  {'cover_image': 'TODO - s3 link', 'epoch': 1498095000000},
  {'cover_image': 'TODO - s3 link', 'epoch': 1498181400000}]
}

With pandas , I tried doing

In [89]: df = pandas.DataFrame(data)

In [90]: df.index = pandas.to_datetime(df['epoch'], unit='ms')

In [103]: df['label'] = df.index.week

In [104]: df
Out[104]: 
                        cover_image          epoch  label
epoch                                                    
2017-06-14 01:30:00  TODO - s3 link  1497403800000     24
2017-06-15 01:30:00  TODO - s3 link  1497490200000     24
2017-06-16 01:30:00  TODO - s3 link  1497576600000     24
2017-06-17 01:30:00  TODO - s3 link  1497663000000     24
2017-06-18 01:30:00  TODO - s3 link  1497749400000     24
2017-06-19 01:30:00  TODO - s3 link  1497835800000     25
2017-06-20 01:30:00  TODO - s3 link  1497922200000     25
2017-06-21 01:30:00  TODO - s3 link  1498008600000     25
2017-06-22 01:30:00  TODO - s3 link  1498095000000     25
2017-06-23 01:30:00  TODO - s3 link  1498181400000     25

In [106]: df.groupby('label').groups
Out[106]: 
{24: DatetimeIndex(['2017-06-14 01:30:00', '2017-06-15 01:30:00',
                '2017-06-16 01:30:00', '2017-06-17 01:30:00',
                '2017-06-18 01:30:00'],
               dtype='datetime64[ns]', name=u'epoch', freq=None),
 25: DatetimeIndex(['2017-06-19 01:30:00', '2017-06-20 01:30:00',
                '2017-06-21 01:30:00', '2017-06-22 01:30:00',
                '2017-06-23 01:30:00'],
               dtype='datetime64[ns]', name=u'epoch', freq=None)}

As my knowledge of pandas is limited, I couldn't go further.

It would be really great if I get the week numbers keys are changed to this_week, next_week and future.

Please help me out.

Answer 1

It seems you need:

df = pd.DataFrame(data)
df.index = pd.to_datetime(df['epoch'], unit='ms')

d = dict(tuple(df.groupby(df.index.week)))

print (d[24])
                        cover_image          epoch
epoch                                             
2017-06-14 01:30:00  TODO - s3 link  1497403800000
2017-06-15 01:30:00  TODO - s3 link  1497490200000
2017-06-16 01:30:00  TODO - s3 link  1497576600000
2017-06-17 01:30:00  TODO - s3 link  1497663000000
2017-06-18 01:30:00  TODO - s3 link  1497749400000

EDIT:

data = [{'cover_image': 'TODO - s3 link', 'epoch': 1497403800000},
 {'cover_image': 'TODO - s3 link', 'epoch': 1497490200000},
 {'cover_image': 'TODO - s3 link', 'epoch': 1497576600000},
 {'cover_image': 'TODO - s3 link', 'epoch': 1497663000000},
 {'cover_image': 'TODO - s3 link', 'epoch': 1497749400000},
 {'cover_image': 'TODO - s3 link', 'epoch': 1497835800000},
 {'cover_image': 'TODO - s3 link', 'epoch': 1497922200000},
 {'cover_image': 'TODO - s3 link', 'epoch': 1498008600000},
 {'cover_image': 'TODO - s3 link', 'epoch': 1498895000000},
 {'cover_image': 'TODO - s3 link', 'epoch': 1499881400000}]

df = pd.DataFrame(data)
df.index = pd.to_datetime(df['epoch'], unit='ms')
print (df)
                        cover_image          epoch
epoch                                             
2017-06-14 01:30:00  TODO - s3 link  1497403800000
2017-06-15 01:30:00  TODO - s3 link  1497490200000
2017-06-16 01:30:00  TODO - s3 link  1497576600000
2017-06-17 01:30:00  TODO - s3 link  1497663000000
2017-06-18 01:30:00  TODO - s3 link  1497749400000
2017-06-19 01:30:00  TODO - s3 link  1497835800000
2017-06-20 01:30:00  TODO - s3 link  1497922200000
2017-06-21 01:30:00  TODO - s3 link  1498008600000
2017-07-01 07:43:20  TODO - s3 link  1498895000000
2017-07-12 17:43:20  TODO - s3 link  1499881400000

---

now = pd.datetime.now()
print (now)
2017-06-14 09:45:25.371940

weeks = df.index.week
this_week = now.isocalendar()[1]
next_week = (now + pd.Timedelta(7, unit='d')).isocalendar()[1]
map_d = {x:'future' for x in weeks.unique() if x not in [this_week, next_week]}
map_d[this_week] = 'this_week'
map_d[next_week] = 'next_week'
print (map_d)
{24: 'this_week', 25: 'next_week', 26: 'future', 28: 'future'}

d = dict(tuple(df.groupby([map_d[x] for x in weeks])))

print (d['next_week'])
                        cover_image          epoch
epoch                                             
2017-06-19 01:30:00  TODO - s3 link  1497835800000
2017-06-20 01:30:00  TODO - s3 link  1497922200000
2017-06-21 01:30:00  TODO - s3 link  1498008600000

---

d = {k:v.to_dict(orient='records') for k, v in df.groupby([map_d[x] for x in weeks])}
print (d)
{'future': [{'cover_image': 'TODO - s3 link', 'epoch': 1498895000000}, 
            {'cover_image': 'TODO - s3 link', 'epoch': 1499881400000}], 
'next_week': [{'cover_image': 'TODO - s3 link', 'epoch': 1497835800000}, 
              {'cover_image': 'TODO - s3 link', 'epoch': 1497922200000}, 
              {'cover_image': 'TODO - s3 link', 'epoch': 1498008600000}], 
 'this_week': [{'cover_image': 'TODO - s3 link', 'epoch': 1497403800000}, 
               {'cover_image': 'TODO - s3 link', 'epoch': 1497490200000}, 
               {'cover_image': 'TODO - s3 link', 'epoch': 1497576600000}, 
               {'cover_image': 'TODO - s3 link', 'epoch': 1497663000000}, 
               {'cover_image': 'TODO - s3 link', 'epoch': 1497749400000}]}