熊猫通过groupby日期的总和归一化由datetimeindex索引的列

Question

If given a dataframe that's indexed with a datetimeindex, is there an efficient way to normalize the values within a given day? For example I'd like to sum all values for each day, and then divide each columns values by the resulting sum for the day.

I can easily group by date and calculate the divisor (sum of values of each column for each date) but I'm not entirely sure the best way to divide the original dataframe by the resulting sum df.

Example dataframe with datetimeindex and resulting df from sum

I attempted to do something like

df / df.groupby(df.index.to_period('D')).sum()

however it isn't behaving as I would have hoped for.

Instead I'm getting a df with NaN everywhere and Date appended as new indexes.

ie Results from above division

Toy recreation:

df = pd.DataFrame([[1,2],[3,4],[5,6],[7,8]],columns=['a','b'], 
              index=pd.to_datetime(['2017-01-01 14:30:00','2017-01-01 14:31:00', 
                                    '2017-01-02 14:30:00', '2017-01-02 14:31:00']))
df / df.groupby(df.index.to_period('D')).sum()

results in

                     a  b
2017-01-01 14:30:00 NaN NaN
2017-01-01 14:31:00 NaN NaN
2017-01-02 14:30:00 NaN NaN
2017-01-02 14:31:00 NaN NaN
2017-01-01  NaN NaN
2017-01-02  NaN NaN

Answer 1

You will need to copy and paste your dataframe as text and not an image so I can help further but here is an example:

sample df

df1 = pd.DataFrame(np.random.randn(5,5), columns=list('ABCDE'),
                  index=pd.date_range('2017-01-03', '2017-01-07'))

df2 = pd.DataFrame(np.random.randn(5,5), columns=list('ABCDE'),
                  index=pd.date_range('2017-01-03', '2017-01-07'))

df = pd.concat([df1,df2])

               A            B          C            D           E
2017-01-03  1.393874    1.933301    0.215026    -0.412957   -0.293925
2017-01-04  0.825777    0.315449    2.317292    -0.347617   -2.427019
2017-01-05  -0.372916   -0.931185   0.049707    0.635828    -0.774566
2017-01-06  1.564714    -1.582461   1.455403    0.521305    -2.175344
2017-01-07  1.255747    1.967338    -0.766391   -0.021921   0.672704
2017-01-03  0.620301    -1.521681   -0.352800   -1.394239   -1.206983
2017-01-04  -0.041829   -0.870871   -0.402440   0.268725    1.499321
2017-01-05  -1.098647   1.690136    1.004087    0.304037    1.235310
2017-01-06  0.305645    -0.327096   0.280591    -0.476904   1.652096
2017-01-07  1.251927    0.469697    0.047694    1.838995    -0.258889

then what you are currently doing:

df / df.groupby(df.index).sum()

                A           B           C          D            E
2017-01-03  0.692032    4.696817    -1.560723   0.228507    0.195831
2017-01-03  0.307968    -3.696817   2.560723    0.771493    0.804169
2017-01-04  1.053357    -0.567944   1.210167    4.406211    2.616174
2017-01-04  -0.053357   1.567944    -0.210167   -3.406211   -1.616174
2017-01-05  0.253415    -1.226937   0.047170    0.676510    -1.681122
2017-01-05  0.746585    2.226937    0.952830    0.323490    2.681122
2017-01-06  0.836585    0.828706    0.838369    11.740853   4.157386
2017-01-06  0.163415    0.171294    0.161631    -10.740853  -3.157386
2017-01-07  0.500762    0.807267    1.066362    -0.012064   1.625615
2017-01-07  0.499238    0.192733    -0.066362   1.012064    -0.625615

Take a look at the first row col A

1.393874 / (1.393874 + 0.620301) = 0.6920322216292031 so your example of df / df.groupby(df.index).sum() is working as expected.

Also be careful if your data contains NaNs because np.nan / a number = nan

update per comment:

df = pd.DataFrame([[1,2],[3,4],[5,6],[7,8]],columns=['a','b'], 
              index=pd.to_datetime(['2017-01-01 14:30:00','2017-01-01 14:31:00', 
                                    '2017-01-02 14:30:00', '2017-01-02 14:31:00']))


# create multiindex with level 1 being just dates
df.set_index(df.index.floor('D'), inplace=True, append=True)

# divide df by the group sum matching the index values of level 1
df.div(df.groupby(level=1).sum(), level=1).reset_index(level=1, drop=True)

                          a         b
2017-01-01 14:30:00   0.250000  0.333333
2017-01-01 14:31:00   0.750000  0.666667
2017-01-02 14:30:00   0.416667  0.428571
2017-01-02 14:31:00   0.583333  0.571429