Explanation of keras layers behavior in tensorflow 2.0

Question

I made an experiment but I cannot understand why I get that result:

1. I create and fit a simple model using functional API with 3 hidden layers:

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

input_ = layers.Input(shape=(X_train.shape[1:]))
hidden1 = layers.Dense(30, activation='relu')(input_)
hidden2 = layers.Dense(30, activation='relu')(hidden1)
hidden3 = layers.Dense(30, activation='relu')(hidden2)
output = layers.Dense(1)(hidden3)
model1 = keras.Model(inputs=input_, outputs=output)
model1.compile(optimizer='adam', loss='mse')
history = model1.fit(x=X_train, y=y_train, epochs=200)

2. Now say I want to investigate the activation of the second hidden layer. I then create another model like this:

model2 = keras.Model(inputs=input_, outputs=hidden2)

Surprisingly for me if I now call model2.weights these weights are the same as model1.layers[2].weights that has been fitted previously, but I never call .fit() on model2 .

This led me to think that the layer weights must be stored in the hidden2 object after I call model1.fit(x=X_train, y=y_train, epochs=200) , so when I define model2 = keras.Model(inputs=input_, outputs=hidden2) it already knows the weights.

But when I try to investigate the hidden2 object I could not find any way to extract the weights.

Hence my question is how is it possible that model2 knows the weights of the model1 if they are not contained in the hidden2 object?

Many thanks

Answer 1

You are using the Functional API, and this is exactly the intended behavior. This API can be used to share weights between layer instances (tensors).

What is happening is that the layer ( layers.Dense(30, activation='relu') ) is the one that stores the layers, and when you generate new tensors (by calling Model(....) ), the weights are reused as you are reusing the actual layers in the model.

If you want for the layer to have new weights, then you should create new layers, not reuse the ones in the model. Otherwise, it is working fine.