解决numpy / scipy中的3D最小二乘

Question

For some integer K around 100, I have 2 * K (n, n) arrays: X_1, ..., X_K and Y_1, ..., Y_K .

I would like to perform K least squares simultaneously, ie find the n by n matrix A minimizing the sum of squares over k: \\sum_k norm(Y_k - A.dot(X_k), ord='fro') ** 2 ( A must not depend on k ).

I am looking for an easy way to do this with numpy or scipy. I know the function I want to minimize is a quadratic form in A so I could do it by hand, but I'm looking for an off-the-shelf way of doing it. Is there one?

Answer 1

Something like this works if n is a small number.

import numpy as np
from scipy.optimize import minimize

K = 5
n = 10

X = np.random.random_sample((K, n, n))
Y = np.random.random_sample((K, n, n))

def opt(A):

    A = np.reshape(A, (n, n))

    # maybe need to transpose X.dot(a) ?
    # if axis is a 2-tuple, it specifies the axes that hold 2-D matrices, 
    # and the matrix norms of these matrices are computed.
    return np.sum(np.linalg.norm(Y - X.dot(A), ord='fro', axis=(1, 2)) ** 2.0)

A_init = np.random.random_sample((n, n))
print(minimize(opt, A_init ))

Careful: The optimization algorithms used by minimize as default are local.

Answer 2

I can't help with the python, but here is the mathematical solution, in case it helps. We seek to minimise

E = Sum { Tr (Y[j]-A*X[j])*(Y[j]-A*X[j])'}

Some algebra yields

E = Tr(P-A*Q'-Q*A'+A*R*A')
where
P = Sum{ Y[j]*Y[j]'}
Q = Sum{ Y[j]*X[j]'}
R = Sum{ X[j]*X[j]'}

If R is invertible a little more algebra yields

E = Tr( (A-Q*S)*R*(A-Q*S)') + Tr( P - Q*S*Q')
where S = inv( R)

Since

(A-Q*S)*R*(A-Q*S)' is positive definite, 

we minimise E by taking A = Q*S.

In this case an algorithm would be:

compute Q
compute R
solve A*R = Q for A (eg by finding the cholesky factors of R)

If R is not invertible, we should use the generalised inverse for S instead of the plain inverse.

Answer 3

In fact the answer was simple, I just needed to create bigger matrices Y and X by horizontally stacking the Y_k (to create Y) and the X_k (to create X). Then I can just solve a regular 2d least squares problem: minimize norm(Y - A.dot(X))