Define loss function in keras using itertools

Question

I want to define a loss function, which represent the distance between the hidden layer output points. Firstly ,I wrote this without keras

import numpy as np
import itertools
pts = np.array([
    [10,10,10],
    [10,11,20],
    [20,11,30],
    [20,10,10],
    [10,10,20],
    ])
diff = list(itertools.combinations(pts, 2))

ptdiff = lambda (p1,p2): (np.sqrt(np.sum((p1 - p2) ** 2)))
diffs = map(ptdiff, diff)
np.mean(diffs)

I get the result. And I try this loss function in keras, z is the output of hidden layer, which is a matrix

define loss function

def vae_loss(z):
    z_diff = list(itertools.combinations(z,2))
    ptdiff = lambda (p1,p2): (np.sqrt(np.sum((p1 - p2) ** 2)))
    z_diffs = map(ptdiff, z_diff)
    loss = K.mean(z_diffs)
    return loss

But it shows the TypeError: 'Tensor' object is not iterable. , I just wonder how can I fix this problem.

Answer 1

Based on this very helpful question, you can make use of Keras' broadcasting properties. I'm assuming here you run Keras on the TensorFlow backend. From the TF docs on broadcasting:

A special case arises, and is also supported, where each of the input arrays has a degenerate dimension at a different index. In this case, the result is an "outer operation".

A reproducible example of your numpy code is the following:

import numpy as np
import itertools

# Generate 100 random points in a 5-D space
n_dim = 5
matrix = np.random.rand(1000, 5)

# List all possible combinations
combinations = list(itertools.combinations(matrix.tolist(), 2))

def mse(tup):
    """MSE between first and second element of a tuple of lists"""
    return np.mean((np.array(tup[0]) - np.array(tup[1]))**2)

avg_mse = np.mean([mse(c) for c in combinations])
print('Average mse: {:.3f}'.format(avg_mse))

This returns, in my case, Average mse: 0.162

Based on the question cited above, you could construct your loss function as follows:

import keras.backend as K

# Wrap our random matrix into a tensor
tensor = K.constant(value=matrix)

def loss_function(x):
    x_ = K.expand_dims(tensor, axis=0)
    x__ = K.expand_dims(tensor, axis=1)

    # Compute mse for all combinations, making use of broadcasting
    z = K.mean(K.square(x_ - x__), axis=-1)

    # Return average mse
    return(K.mean(z))

with K.get_session() as sess:
    print('Average mse: {:.3f}'.format(loss_function(tensor).eval()))

Which returns for me Average mse: 0.162 .

Note that this implementation does not exactly replicate the behaviour from your numpy example. The difference is that all combinations of rows with itself are also considered (not the case with itertools.combinations ) and combinations are considered twice: mse((row1, row2)) and mse((row2, row1)) will both be calculated, which is again not the case with your itertools code. For matrices with a large number of rows, this should not make too much of a difference, as my example shows.