内部 Keras model 中的多个嵌入层的问题

Question

I am trying to construct a Keras model model_B that outputs the output of another Keras model model_A . Now, the output of model_A is computed from the concatenation of several tensors coming from multiple Keras embedding layers with different vocabulary sizes. Models model_A and model_B are essentially the same.

Problem: When I train model_A , everything works fine. However, when I train model_B on the same dataset, I get the following error:

tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[1] = 3 is not in [0, 2) [[{{node model_1/embedding_1/embedding_lookup}}]]

Essentially, the error is saying that the index of a word is outside of the expected vocabulary, but this is not the case. Could someone clarify why this happens?

---

Here is a reproducible example of the problem:

from keras.layers import Input, Dense, Lambda, Concatenate, Embedding
from keras.models import Model
import numpy as np


# Constants
A = 2
vocab_sizes = [2, 4]

# Architecture
X = Input(shape=(A,))
embeddings = []
for a in range(A):
    X_a = Lambda(lambda x: x[:, a])(X)
    embedding = Embedding(input_dim=vocab_sizes[a],
                          output_dim=1)(X_a)
    embeddings.append(embedding)
h = Concatenate()(embeddings)
h = Dense(1)(h)

# Model A
model_A = Model(inputs=X, outputs=h)
model_A.compile('sgd', 'mse')

# Model B
Y = Input(shape=(A,))
model_B = Model(inputs=Y, outputs=model_A(Y))
model_B.compile('sgd', 'mse')

# Dummy dataset
x = np.array([[vocab_sizes[0] - 1, vocab_sizes[1] - 1]])
y = np.array([1])

# Train models
model_A.fit(x, y, epochs=10)  # Works well
model_B.fit(x, y, epochs=10)  # Fails

From the error above, it somehow seems that the input x[:, 1] is wrongly being fed to the first embedding layer with vocabulary size 2, as opposed to the second. Interestingly, when I swap the vocabulary sizes (eg set vocab_sizes = [4, 2] ) it works, supporting the previous hypothesis.

Answer 1

For some weird reason, looping the tensor is causing this error. You can replace your slicing with tf.split , use the necessary adjusts and it will work well:

Extra imports:

import tensorflow as tf
from keras.layers import Flatten

# Architecture
X = Input(shape=(A,))
X_as = Lambda(lambda x: tf.split(x, A, axis=1))(X)

embeddings = []
for a, x in enumerate(X_as):
    embedding = Embedding(input_dim=vocab_sizes[a],
                          output_dim=1)(x)
    embeddings.append(embedding)
h = Concatenate(axis=1)(embeddings)
h = Flatten()(h)
h = Dense(1)(h)

Why does this happen?

Well, it's very hard to guess. My assumption is that the system is trying to apply the lambda layer using the actual variable a instead of the value you gave before (this should not be happenning, I guess, but I had exatly this problem once when loading a model: one of the variables kept its last value when loading the model instead of having a looped value)

One thing that supports this explanation is trying constants instead of a :

#Architecture
X = Input(shape=(A,))
embeddings = []

X_a1 = Lambda(lambda x: x[:, 0], name = 'lamb_'+str(0))(X)
X_a2 = Lambda(lambda x: x[:, 1], name = 'lamb_'+str(1))(X)
xs = [X_a1, X_a2]

for a, X_a in enumerate(xs):
    embedding = Embedding(input_dim=vocab_sizes[a],
                          output_dim=1)(X_a)
    embeddings.append(embedding)
h = Concatenate()(embeddings)
h = Dense(1)(h)

Solution if you want to avoid tf.split

Another thing that works (and supports the explanation that the Lambda might be using the last value of a in your code for model_B ) is making the entire loop inside the Lambda layer, this way, a doesn't get any unexpected values:

#Architecture
X = Input(shape=(A,))
X_as = Lambda(lambda x: [x[:, a] for a in range(A)])(X)

embeddings = []
for a, X_a in enumerate(X_as):
    embedding = Embedding(input_dim=vocab_sizes[a],
                          output_dim=1)(X_a)
    embeddings.append(embedding)
h = Concatenate()(embeddings)
h = Dense(1)(h)

Answer 2

I believe the following is happening:

(1) When you do the initial "for loop" over the Lambda function, you are initializing the constant tensors which feed into the "strided_slice" operator that extracts either the [:,0] or [:,1] correctly. Using the global variable "a" in the Lambda function is probably "risky" but works okay in this instance. Furthermore, I believe that the function is being stored in bytecode as "lambda x: x[:, a]" so it will try to look up whatever the value of "a" is at the time of evaluation. "a" could be anything so might be problematic under some cases.

(2) When you build the first model (model_A), the constant tensors are not reinitialized, so the lambda functions (strided_slice operator) has the correct values (0 and 1) which were initialized in the "for loop."

(3) When you build the second model (model_B), the constant tensors are reinitialized. However, at this time, the value of "a" is now 1 (as stated by some of the other commentary), because that is the final value after the original "for loop." In fact, you can set a=0, just before defining model_B, and you'll actually get behavior which corresponds to both Lambdas extracting [:,0] and feeding it to the embedded layers. My speculation for this difference in behavior is perhaps related to calling the Model_A(X) class initialization in this case (whereas in the first model, you only specified the output layer "h" and didn't call the Model_A() class as the output - this difference I believe was also suggested by other commentary).

I'll say that I verified this state of affairs by putting in some print statements in the file "frameworks/constant_op.py" during the operator initialization step and obtained debug statements with values and sequences consistent with what I stated above.

I hope this helps.

Answer 3

You shouldn't call Model_A(X) directly. It will return a tensor, you can call model_B = Model(inputs=X, outputs=model_A.outputs) . It works for me.

The difference between Model_A(X) and Model_A.outputs is:

Model_A(X) # a tensor
# <tf.Tensor 'model_13/dense_8/BiasAdd:0' shape=(?, 1) dtype=float32>

Model_A.outputs # a list of tensor
# [<tf.Tensor 'dense_8/BiasAdd:0' shape=(?, 1) dtype=float32>]