在 dataframe 中循环，日期和日期之间在 python

Question

I would like to process all the data between two dates changing the dates. In particular, I have the following dataframe:

                   real    model2      model1
date                                               
2017-01-01 00:00:00   51.22   52.776425   52.583711
2017-01-01 01:00:00   53.00   47.211506   50.679937
2017-01-01 02:00:00   52.00   44.722529   48.478772
2017-01-01 03:00:00   51.00   42.475170   45.141708
2017-01-01 04:00:00   47.27   38.862827   44.583250
2017-01-01 05:00:00   45.49   39.473972   44.930338
2017-01-01 06:00:00   45.69   42.465659   47.380179

where dates are also indexes. I would like to collect all the data day by day in a list to pass to a function. I have done it in a not smart\correct way as:

for iday in range(1,9):
   #
   #
   start_date = '2017-01-0'+str(iday)+ ' 00:00:00'
   end_date   = '2017-01-0'+str(iday)+ ' 23:00:00'
   #
   data_sub_e = EE.loc[start_date:end_date]

It sounds not correct, it is difficult to extend to a number of day greater then 10 and it seems to not use pandas feature.

Is there any smart way to do that?

Thanks in advance,

Diego

Answer 1

I assume that date is of datetime type (not string ).

Using df.index.date you can select rows by the date part only .

Eg:

d1 = pd.to_datetime('2017-01-01')  # The criterion date
df[df.index.date == d1]   # Get all rows from this date, whatever the hour part

Another hint : Instead of your loop based on the day number:

for iday in range(1,9):

run a loop based on pd.date_range , something like:

for dat in pd.date_range('2017-01-01', '2017-01-15', freq='D'):

Of course, set the end date according to your needs.

Another choice can be to group your DataFrame by the date part of the index:

df.groupby(pd.Grouper(freq='D'))

and then apply your function to each group.

Edit following the comment

To change your values into lists, for each group, you can use named aggregation :

df.groupby(pd.Grouper(freq='D')).agg({'real': list,
    'model1': list, 'model2': list})

If you want to assign own column names, you can use another syntax, with named parameters:

df.groupby(pd.Grouper(freq='D')).agg(Real=('real', list),
    Model_1=('model1', list), Model_2=('model2', list))

Here parameter names specify output column names. The value of each parameter is a tuple: ( original column name , aggregation function ).