Saving and loading a keras model

Question

I have trained a model which has among others 2 embeddings which have initial weights pre-trained (using gensim). Both embeddings have trainable=True meaning that the weights are updated in the training. The model is something like the following:

def init_embedding(emb_matrix, name):
    embedding_layer = Embedding(
        emb_matrix.shape[0],
        emb_matrix.shape[1],
        weights=[emb_matrix],
        trainable=True,
        name=name)
    return embedding_layer

embedding_layer1 = init_embedding(gensimWord2vecMatrix, "emb_1")
embedding_layer2 = init_embedding(gensimWord2vecMatrix2, "emb_2")


sequence_input_1 = Input(shape=(10,), dtype='int32', name="input_1")
sequence_input_2 = Input(shape=(1,), dtype='int32', name="input_2")
sequence_input_3 = Input(shape=(100,), dtype='int32', name="input_3")


embedded_sequences_1 = embedding_layer1(sequence_input_1)
embedded_sequences_2 = embedded_layer2(sequence_input_1)
embedded_sequences_3 = embedded_layer2(sequence_input_3)

conv_step1 = Convolution1D(filters=1000,kernel_size=5,activation=activation_fun,name="conv_layer_",padding="valid")(embedded_sequences_1)
        .
        .

z = Dense(ashape, activation="linear", name="predicted_description")(conv_step10)
loss = a_loss([z, embedded_sequences_2, embedded_sequences_3])# a loss function
model = Model(
    inputs=[sequence_input_1, sequence_input_2, sequence_input_],
    outputs=[loss])
    model.compile(loss=mean_loss, optimizer=Adam()) 

and what I am trying to do is to save/load the model. The first that I tried was to:

model.save("./test_model.h5")

But after when I use:

from keras.models import load_model
model = load_model("./test_model.h5")

I got an error of : ValueError: Missing layer: x

Searching through the internet a solution is to save only the weights and define again the model (without the compile ) and assign the saved weights. Something like:

model.save_weights("./model_weights.h5")

and again difining the model without the model.compile in the end:

embedding_layer1 = init_embedding(gensimWord2vecMatrix, "emb_1")
embedding_layer2 = init_embedding(gensimWord2vecMatrix2, "emb_2")
         .
         .
         .
model = Model(...
model.load_weights("./test_model.h5")

which seems to work fine and I can use the predict function. My question is what happens with the trainable embeddings using this methodology. Are the predict functions using the trained weights from the model or it using the weights that are initialized from the gensim model? because to use the load_weights as shown above I am recreating the embeddings from the beginning.

Any idea? I hope my question was clear. Any comment is useful for restructuring the question if its not understandable.

Answer 1

If you create the embeddings first and later you load weights, the loaded weights will prevail (as long as you never call init embedding after that).

You can also make a manual check.

Create the model, get each embedding layer weights:

w1 = model.get_layer('embeddingName1').get_weights()
#same for 2 and 3

Load the weights and get them again from the layers:

newW1 = model.get_layer('embeddingName1').get_weights()

Compare them:

for w,newW in zip(w1,newW1):
    print((w==newW).all()) #if false, weights have changed with loading   

#same for 2 and 3.