沿第三维在 Python 中取百分位数

Question

I've been struggling with this one for a bit now. I have a matrix that is 55115 x 34, where each number along the first dimension is one day, for 151 years, totally 55115 points.

I am trying to get monthly percentiles of the values in the first dimension, so I have first added a date column, which subsequently groups it into months, although I cannot figure out the best way to take the percentile (95th) of both the days and the third dimension (here is 34). So after grouping the months, the matrix should be 151 x 12 x 34, and I want to take the 95th percentile along the third dimension, so my final matrix would be 151 x 12, in theory. Below is what I have so far to add the dates to the array:

dates = pd.date_range(start='1950-01-01', end='2100-12-31', freq='D') #create daily date range from 1950 to 2100

leap = [] #empty array
for each in dates:
    if each.month==2 and each.day ==29: #find each leap day (feb 29)
        leap.append(each)

dates = dates.drop(leap) #get rid of leap days
dates = pd.to_datetime(dates) #convert to datetime format 
data = {'wind': winddata, 'time': dates} #create table with both dates and data
df = pd.DataFrame(data) #create dataframe
df.set_index('time') #index time
df.groupby(df['time'].dt.strftime('%b'))['wind'].sort_values()

And this is what I have to take the percentile:

months = df.groupby(pd.Grouper(key='time',freq = "M")) #group each month
monthly_percentile = months.aggregate(lambda x: np.percentile(x, q = 95)) #percentile across each month 

Although, this does not appear to work. I'm open to other methods of doing this, I just am hoping to a) rearrange the 55115 x 34 data set into months, so that it is 151 (years) x 365 (days) x 34 (ensembles), and then the percentile is taken across the months and third dimension so I end up with 151 x 12 total. I'm happy to clarify anything if I did not specify well enough. Any detailed response would be really helpful. Thank you so much in advance!

Answer 1

If I get your question right, the most straightforward solution I can think of is to add the columns year and month , then groupby over them and compute a required percentile:

import pandas as pd
import numpy as np

dates = pd.date_range(start='1950-01-01', end='2100-12-31', freq='D')
dates_months = [date.month for date in dates]
dates_years = [date.year for date in dates]
values = np.random.rand(34, len(dates))
df = pd.DataFrame()

df['date'] = dates
df['year'] = dates_years
df['month'] = dates_months
for i in range(34):
    df[f'values_{i}'] = values[i]

df = df.melt(id_vars=['date', 'year', 'month'], value_vars=[f'values_{i}' for i in range(34)])
sub = df.groupby(['year', 'month']).value.apply(lambda x: np.quantile(x, .95)).reset_index()

finally, if you really need a 151 x 12 array instead of year-month-percentile table of length 1812 (=151*12) you could use something like this:

crosstab = pd.crosstab(index=sub['year'], columns=sub['month'], values=sub['values'], aggfunc=lambda x: x)