将函数应用于每个子集组合并返回平方矩阵

Question

I do not know how to do this without four nested for loops.

I'd like to apply a custom function to every possible combination of subsets for hour and day , return that value, and then pivot the data frame into a square matrix. However, these for loops seem excessive so I'm looking for a more efficient way to do this. The data I have is fairly large so any gain in speed would be beneficial.

edit: I updated the question to include a custom function.

Here is an example,

Sample data

import pandas as pd
import numpy as np
dat = pd.DataFrame({'day': {0: 1, 1: 1, 2: 1, 3: 1, 4: 1, 5: 1, 6: 1, 7: 1, 8: 1, 9: 1, 10: 2, 11: 2, 12: 2, 13: 2, 14: 2, 15: 2, 16: 2, 17: 2, 18: 2, 19: 2}, 'hour': {0: 1, 1: 1, 2: 1, 3: 1, 4: 1, 5: 2, 6: 2, 7: 2, 8: 2, 9: 2, 10: 1, 11: 1, 12: 1, 13: 1, 14: 1, 15: 2, 16: 2, 17: 2, 18: 2, 19: 2}, 'distance': {0: 1.2898851269657656, 1: 0.0, 2: 0.8371526423804061, 3: 0.8703856587273138, 4: 0.6257425922449789, 5: 0.0, 6: 0.0, 7: 0.0, 8: 1.2895328696587023, 9: 0.0, 10: 0.6875527848294374, 11: 0.0, 12: 0.0, 13: 0.9009031833559706, 14: 0.0, 15: 1.1040652963428623, 16: 0.0, 17: 0.0, 18: 0.0, 19: 0.0}})

Code

def custom_fn(x, y):
    x = pd.Series(x)
    y = pd.Series(y)
    x = x**2
    y = np.sqrt(y)
    return x.sum() - y.sum()

# Empty data.frame to append to
dmat = pd.DataFrame()

# For i, j = hour; k, l = day
for i in range(1, 3):
    for j in range(1, 3):
        for k in range(1, 3):
            for l in range(1, 3):
                x = dat[(dat['hour'] == i) & (dat['day'] == k)].distance
                y = dat[(dat['hour'] == j) & (dat['day'] == l)].distance
                # Calculate difference
                jds = custom_fn(x, y)
                # Build data frame and append
                outdat = pd.DataFrame({'day_hour_a': f"{k}_{i}", 'day_hour_b': f"{l}_{j}", 'jds': [round(jds, 4)]})
                dmat = dmat.append(outdat, ignore_index=True)

# Pivot data to get matrix
distMatrix = dmat.pivot(index='day_hour_a', columns='day_hour_b', values='jds')

Output

> print(distMatrix)

day_hour_b     1_1     1_2     2_1     2_2
day_hour_a                                
1_1        -0.2609  2.3782  1.7354  2.4630
1_2        -2.1118  0.5273 -0.1155  0.6121
2_1        -2.4903  0.1488 -0.4940  0.2336
2_2        -2.5557  0.0834 -0.5594  0.1682

Answer 1

If I understand correctly, what you're doing is the same as the following:

def f(x):
    return x.mean()

x = df.groupby(['day', 'hour'])['distance'].apply(f)
x = x.values[:,None] - x.values

print(x)

Output:

[[ 0.          0.46672663  0.40694201  0.50382014]
 [-0.46672663  0.         -0.05978462  0.03709351]
 [-0.40694201  0.05978462  0.          0.09687813]
 [-0.50382014 -0.03709351 -0.09687813  0.        ]]

Update: For the updated custom function you can still break it down into separate groupby s:

g = df.groupby(['day', 'hour'])['distance']
x = g.apply(lambda z: (z**2).sum())
y = g.apply(lambda z: np.sqrt(z).sum())

x.values[:,None] - y.values

Output:

array([[-0.26092193,  2.37817717,  1.73540595,  2.46300806],
       [-2.11178008,  0.52731901, -0.1154522 ,  0.61214991],
       [-2.49031973,  0.14877937, -0.49399185,  0.23361026],
       [-2.55571493,  0.08338417, -0.55938705,  0.16821506]])

Update 2: If calculations can't be separated, another alternative is:

def f(x, y):
    return distance.jensenshannon(x, y)

x = []
g = df.groupby(['day', 'hour'])['distance']
for k1, g1 in g:
    for k2, g2 in g:
        x += [(k1, k2, f(g1, g2))]

x = pd.DataFrame(x).pivot(index=0, columns=1, values=2)

print(x)

Output:

1         (1, 1)    (1, 2)    (2, 1)    (2, 2)
0                                             
(1, 1)  0.000000  0.623167  0.419371  0.550291
(1, 2)  0.623167  0.000000  0.424608  0.832555
(2, 1)  0.419371  0.424608  0.000000  0.504233
(2, 2)  0.550291  0.832555  0.504233  0.000000