Groupby + DataFrame与系列之间的关联

Question

I have a DataFrame a and Series b . I want to find conditional correlation of each column of a to b , conditional on the value of b . Specifically, I'm using pd.cut to break up b into 5 groups. But instead of a standard quantile, I'm using standard deviations of b above or below the mean.

np.random.seed(123)

a = (pd.DataFrame(np.random.randn(1000,3))
     .add_prefix('col'))
b = pd.Series(np.random.randn(1000))

mu, sigma = b.mean(), b.std()
breakpoints = mu + np.array([-2., -1., 1., 2.]) * sigma
breakpoints = np.append(np.insert(breakpoints, 0, -np.inf), np.inf)
# There are now 6 breakpoints to create 5 groupings:
# array([       -inf, -1.91260048, -0.9230609 ,  1.05601827,  2.04555785,
#                inf])

labels = ['[-inf,-2]', '(-2,-1]', '(-1,1]', '(1,2]', '(2,inf]']
groups = pd.cut(b, bins=breakpoints, labels=labels)

All is good through here. I'm hung up on the final line, using .corrwith with .groupby , which throws a ValueError :

a.groupby(groups).corrwith(b.groupby(groups))

Any ideas? The result of a.corrwith(b) is a Series, so I'm thinking the result here should be a DataFrame with the groups/buckets as columns. For example, one column would be:

print(a[b < breakpoints[1]].corrwith(b[b < breakpoints[1]]))
# Correlation conditional on that `b` is [-inf, -2 stdev]
col0    0.43708
col1   -0.08440
col2   -0.02923
dtype: float64

Answer 1

One solution that's functional but not pretty:

full = a.join(b.to_frame(name='_drop'))
corrs = (full.groupby(groups)
         .corr()
         .loc[(slice(None), a.columns), '_drop']
         .unstack()
         .T)

print(corrs)
      [-inf,-2]  (-2,-1]   (-1,1]    (1,2]  (2,inf]
col0    0.43708  0.06716  0.02437  0.01695  0.05384
col1   -0.08440  0.04208  0.05529 -0.07146  0.14766
col2   -0.02923 -0.19672  0.01519 -0.02290 -0.17101