数据清洗python数据框

Question

I have a Python dataframe with 1408 lines of data. My goal is to compare the largest number and smallest number associated with a given weekday during one week to the next week's number on the same day of the week which the prior largest/smallest occurred. Essentially, I want to look at quintiles (since there are 5 days in a business week) rank 1 and 5 and see how they change from week to week. Build a cdf of numbers associated to each weekday.

1. To clean the data, I need to remove 18 weeks in total from it. That is, every week in the dataframe associated with holidays plus the entire week following week after the holiday occurred.

2. After this, I think I should insert a column in the dataframe that labels all my data with Monday through Friday-- for all the dates in the file (there are 6 years of data). The reason for labeling MF is so that I can sort each number associated to the day of the week in ascending order. And query on the day of the week.

Methodological suggestions on either 1. or 2. or both would be immensely appreciated.

Thank you!

Answer 1

#2 seems like it's best tackled with a combination of df.groupby() and apply() on the resulting Groupby object. Perhaps an example is the best way to explain.

Given a dataframe:

In [53]: df
Out[53]: 
            Value
2012-08-01     61
2012-08-02     52
2012-08-03     89
2012-08-06     44
2012-08-07     35
2012-08-08     98
2012-08-09     64
2012-08-10     48
2012-08-13    100
2012-08-14     95
2012-08-15     14
2012-08-16     55
2012-08-17     58
2012-08-20     11
2012-08-21     28
2012-08-22     95
2012-08-23     18
2012-08-24     81
2012-08-27     27
2012-08-28     81
2012-08-29     28
2012-08-30     16
2012-08-31     50

In [54]: def rankdays(df):
  .....:    if len(df) != 5:
  .....:        return pandas.Series()
  .....:    return pandas.Series(df.Value.rank(), index=df.index.weekday)
  .....: 

In [52]: df.groupby(lambda x: x.week).apply(rankdays).unstack()
Out[52]: 
    0  1  2  3  4
32  2  1  5  4  3
33  5  4  1  2  3
34  1  3  5  2  4
35  2  5  3  1  4