Custom Loss Function for Reward using Keras in Python

Question

I have a model, which I would like to build a custom loss function, I have my States which are my X values and then I have my actions which are 7 one-hot categorical values which are my Y values, that I am predicting.

However I'm not sure how to pass the reward to the loss function. I'm also not sure what the actual function should be, but I can experiment with this later.

x = input_data[:, :-2]  # States
y = input_data[:, -2]  # Actions
r = input_data[:, -1]  # Rewards

def custom_loss(y_pred, y_true):
     loss = K.square(y_pred - y_true) * r
     return loss

model.compile(loss=custom_loss, optimizer='adam', metrics=['accuracy'])
model.fit(x, y)

Answer 1

You can write a function that returns another function. You pass the reward as a parameter to the top function:

def penalized_loss(reward):
  def custom_loss(y_true, y_pred):
    return K.mean(K.square(y_pred - y_true) - K.square(y_true - reward), axis=-1)

  return custom_loss

.
.
.
model.compile(loss=[penalized_loss(reward=r)], optimizer='adam', metrics=['accuracy'])

I am also providing a gist with a very silly working example: https://gist.github.com/kolygri/c222adba4dff710c6c53bf83c0ed5d21