X X^T Matrix is not positive definite, although it should be

Question

I have a matrix M where M.shape = (679, 512) .

I would like to find the eigenvectors and eigenvalues of MM^T , it's covatiance matrix, which should be positive definite in maths. I find them using:

import numpy as np
v, w = np.linalg.eig(np.matmul(M, M.T))

However, for some reason I see my eigenvalues/eigenvectors as:

>>> v
array([ 1.17577206e+10+0.0000000e+00j,  1.05090161e+10+0.0000000e+00j,
        7.01098189e+09+0.0000000e+00j,  5.19451802e+09+0.0000000e+00j, 
    ...
    1.03985971e-12+0.0000000e+00j, -3.04194468e-13+6.9097571e-13j,
   -3.04194468e-13-6.9097571e-13j,  1.77363116e-13+0.0000000e+00j,
   -1.20885913e-13+0.0000000e+00j, -4.95627438e-14+0.0000000e+00j,
   -1.56456859e-16+0.0000000e+00j], dtype=complex64)

How is this possible? Shouldn't MM^T be positive semi definite, giving positive and real eigenvalues only? How is it possible to get non real eigenvalues?

def is_positive_def(x):
    return np.all(np.linalg.eigvals(x) > 0)

Answer 1

"Shouldn't MM^T be positive semi definite, giving positive and real eigenvalues only?" If you change "positive" to "nonnegative", then yes, that is true mathematically. In fact, instead of numpy.linalg.eig , you can use numpy.linalg.eigh , which will return real eigenvalues and eigenvectors.

Note that in a few places, you misstated that claim, and said that MM^T is positive definite. That is not correct. If M has shape (679, 512), then 679 - 512 = 167 of the eigenvalues will be 0; MM^T is semi-definite, not positive definite .

In the numerical computation of the eigenvalues, all those theoretical values of 0 will not be computed exactly by eig or eigh . The first 167 eigenvalues will be close to zero, and numerical error can result in some of those values being negative. When I did the calculation on a random M (specifically, M = np.random.gamma(8, size=(679, 512)) , the largest eigenvalue of MM^T was about 2.227e7, the smallest was -5.822e-10, and the smallest magnitude was 3.881e-13. Note that 5.822e-10/2.227e7 (ie np.abs(v.min()) / v.max() ) is about 2.61e-17, so all those small eigenvalues are effectively 0 relative to the largest eigenvalue .