Keras custom loss function with different weights per example

Question

I'm trying to implement in Keras a custom loss function where each individual example (not class) has a different weight.

To be precise, given the usual y_true (eg. <1,1,0>) and y_pred (eg <1,0.2,0.8>), I'm trying to create weights (eg <0.81, 0.9, 1.0>) and use these with the binary_crossentropy loss function. I have tried:

import numpy as np
from keras import backend as K

def my_binary_crossentropy(y_true, y_pred):
    base_factor = 0.9
    num_examples = K.int_shape(y_true)[0]

    out = [ K.pow(base_factor, num_examples - i - 1) for i in range(num_examples) ]
    forgetting_factors = K.stack(out)

    return K.mean(
        forgetting_factors * K.binary_crossentropy(y_true, y_pred),
        axis=-1
    )

And works fine with this simple example:

y_true = K.variable( np.array([1,1,0]) )
y_pred = K.variable( np.array([1,0.2,0.8]) )
print K.eval(my_binary_crossentropy(y_true, y_pred))

However, when I use it with model.compile(loss=my_binary_crossentropy, ...) I get the following error: TypeError: range() integer end argument expected, got NoneType .

I have tried a few things. I replaced K.int_shape with K_shape and now getting: TypeError: range() integer end argument expected, got Tensor. I further replaced range() with K.arange() and now getting: TypeError: Tensor objects are not iterable when eager execution is not enabled. To iterate over this tensor use tf.map_fn TypeError: Tensor objects are not iterable when eager execution is not enabled. To iterate over this tensor use tf.map_fn .

Can anyone help me please? What am I missing? Many Thanks!

Answer 1

K.pow can take a sequence of exponents as argument. So you can compute the exponents first, as a tensor ( [num_examples - 1, num_examples - 2, ..., 0] ), and then feed this tensor into K.pow . Here num_examples is basically just K.shape(y_pred)[0] , which is also a tensor.

def my_binary_crossentropy(y_true, y_pred):
    base_factor = 0.9
    num_examples = K.cast(K.shape(y_pred)[0], K.floatx())
    exponents = num_examples - K.arange(num_examples) - 1
    forgetting_factors = K.pow(base_factor, exponents)
    forgetting_factors = K.expand_dims(forgetting_factors, axis=-1)
    forgetting_factors = K.print_tensor(forgetting_factors)  # only for debugging

    loss = K.mean(
        forgetting_factors * K.binary_crossentropy(y_true, y_pred),
        axis=-1
    )
    loss = K.print_tensor(loss)  # only for debugging
    return loss

As an example, the output printed by the two K.print_tensor statements would be like:

model = Sequential()
model.add(Dense(1, activation='sigmoid', input_shape=(100,)))
model.compile(loss=my_binary_crossentropy, optimizer='adam')

model.evaluate(np.zeros((3, 100)), np.ones(3), verbose=0)
[[0.809999943][0.9][1]]
[0.56144917 0.623832464 0.693147182]

model.evaluate(np.zeros((6, 100)), np.ones(6), verbose=0)
[[0.590489924][0.656099916][0.728999913]...]
[0.409296423 0.454773813 0.505304217...]

The numbers are not exact due to rounding errors. The forgetting_factors (first lines printed after model.evaluate ) are indeed the powers of 0.9. You can also verify that the returned loss values decay by a factor of 0.9 ( 0.623832464 = 0.693147182 * 0.9 and 0.56144917 = 0.693147182 * 0.9 ** 2 , etc).

Answer 2

In tensorflow you first predefine you graph with tensors before you run it. So it's quite common that a function which works with numpy array won't work with tensorflow. In your case num_examples is the problem.

Imagine that in tensorflow this loss function will not be called every time you need it, instead this loss function will build the graph for calculating the loss function inside your graph when the model is trained.

So when keras wants to try to build your loss function inside tensorflow, your y_true is an abstract tensor which most likely will have None for your first shape because the batch_size isn't defined yet.

You have to rewrite your loss function in a way that you're not dependent on your batch_size => remove the variable num_examples