Python中的Pairwise Kullback Leibler（或Jensen-Shannon）发散距离矩阵

Question

I have two matrices X and Y (in most of my cases they are similar) Now I want to calculate the pairwise KL divergence between all rows and output them in a matrix. Eg:

X = [[0.1, 0.9], [0.8, 0.2]]

The function should then take kl_divergence(X, X) and compute the pairwise Kl divergence distance for each pair of rows of both X matrices. The output would be a 2x2 matrix.

Is already some implementation for this in Python? If not, this should be quite simple to calculate. I'd like some kind of matrix implementation for this, because I have a lot of data and need to keep the runtime as low as possible. Alternatively the Jensen-Shannon entropy is also fine. Eventually this would even be a better solution for me.

Answer 1

Note that KL divergence is essentially a dot product of P(i) and log(P(i)/Q(i)). So, one option is to form a list of numpy arrays for P(i) and another for log(P(i)/Q(i)), one row for each KL divergence you want to calculate), then perform dot-products.

Answer 2

There is a new(ish) library called dit which has JSD implemented, as well as mutual information and many other distance metrics:

import dit
foo = dit.Distribution(['A','B','C'],[0.5,0.5,0.0])
bar = dit.Distribution(['A','B','C'],[0.1,0.0,0.9])
dit.divergences.jensen_shannon_divergence([foo,bar])
0.80499327350549388

The docs could use a bit of work, but it looks promising.

http://docs.dit.io/en/latest/generalinfo.html#quickstart