自定义损失函数，它依赖于keras中的另一个神经网络

Question

I have a "How can I do that" question with keras :

Assuming that I have a first neural network, say NNa which has 4 inputs (x,y,z,t) which is already trained . If I have a second neural network, say NNb, and that its loss function depends on the first neural network.

The custom loss function of NNb customLossNNb calls the prediction of NNa with a fixed grid (x,y,z) and just modify the last variable t.

Here in pseudo-python-code what I would like to do to traine the second NN : NNb:

grid=np.mgrid[0:10:1,0:10:1,0:10:1].reshape(3,-1).T

Y[:,0]=time
Y[:,1]=something

def customLossNNb(NNa,grid):
     def diff(y_true,y_pred): 
         for ii in range(y_true.shape[0]):
               currentInput=concatenation of grid and y_true[ii,0]
               toto[ii,:]=NNa.predict(currentInput)
               #some stuff with toto
         return #...
     return diff

Then

NNb.compile(loss=customLossNNb(NNa,K.variable(grid)),optimizer='Adam')
NNb.fit(input,Y)

In fact the line that cause me troubles is currentInput=concatenation of grid and y_true[ii,0]

I tried to send to customLossNNb the grid as a tensor with K.variable(grid) . But I can't defined a new tensor inside the loss function, something like CurrentY which has a shape (grid.shape[0],1) fill with y[ii,0] ( ie the current t) and then concatenate grid and currentY to build currentInput

Any ideas?

Thanks

Answer 1

You can include your custom loss function into the graph using functional API of keras. The model in this case can be used as a function, something like this:

for l in NNa.layers: 
    l.trainable=False
x=Input(size)
y=NNb(x)
z=NNa(y)

Predict method will not work, since loss function should be part of the graph, and predict method returns np.array

Answer 2

First, make NNa untrainable. Notice that you should do this recursively if your model has inner models.

def makeUntrainable(layer):
    layer.trainable = False

    if hasattr(layer, 'layers'):
        for l in layer.layers:
            makeUntrainable(l)

makeUntrainable(NNa)

Then you have two options:

• Attach NNa to the end of your model (notice that both y_true and y_pred will be changed)
  • Then change your targets (predict with NNa) for correct results since your model is now expecting the output of NNa, not NNb.
• Create a custom loss function that uses NNa inside it, without changing your targets

Option 1 - Attaching models

inputs = NNb.inputs   
outputs = NNa(NNb.outputs) #make sure NNb is outputing 4 tensors to match NNa inputs   
fullModel = Model(inputs,outputs)

#changing the targets:
newY_train = NNa.predict(oldY_train)    

Option 2 - Creating a custom loss

Warning: please test whether NNa's weights are really frozen while training this configuration

from keras.losses import binary_crossentropy

def customLoss(true,pred):
    true = NNa(true)
    pred = NNa(pred)

    #use some of the usual losses or create your own
    binary_crossentropy(true,pred)

NNb.compile(optimizer=anything, loss = customLoss)