关于Keras嵌入层的输入形状的困惑

Question

I'm trying to use the Keras embedding layer to create my own CBoW implementation to see how it works.

I've generated outputs represented by a vector of the context word I'm searching for with size equal to my vocab. I've also generated inputs so that each context word has X many nearby words represented by their one-hot encoded vectors.

So for example if my sentence is:

"I ran over the fence to find my dog"

using window size 2, I could generate the following input/output:

[[over, the, to, find], fence] where 'fence' is my context word, 'over', 'the', 'to', 'find' are my nearby words with window 2 (2 in front, 2 in back).

Using sample vocab size of 500 and 100 training samples, after one-hot encoding my input and output, it would have the following dimensions:

y.shape -> (100,500)
X.shape -> (100,4,500)

That is, I have 100 outputs each represented by a 500-sized vector. I have 100 inputs each represented by a series of 4 500-sized vectors.

I have a simple model defined as:

model = Sequential()
model.add(Embedding(input_dim=vocabulary_size, output_dim=embedding_size, input_length=2*window_size))
#take average of context words at hidden layer
model.add(Lambda(lambda x: K.mean(x, axis = 1), output_shape=(embedding_size,)))
model.add(Dense(vocabulary_size, activation='softmax'))
model.compile(loss = 'categorical_crossentropy', optimizer = 'adam')

However, when I try to fit my model, I get a dimensional exception:

model.fit(X, y, batch_size=10, epochs=2, verbose=1)
ValueError: Error when checking input: expected embedding_6_input to have 2 dimensions, but got array with shape (100, 4, 500)

Now, I can only assume I'm using the embedding layer wrongly. I've read both this CrossValidated Question and the Keras documentation .

I'm still not sure exactly how the inputs of this embedding layer works. I'm fairly certain my input_dim and output_dim are correct, which leaves input_length . According to the CrossValidated, my input_length is the length of my sequence. According to Keras, my input should be of dimension (batch_size, input_length) .

If my inputs are 4 words each represented by a word vector of size vocab_size , how do I input this to the model?

Answer 1

The problem is that you are thinking about the embedding layer in a wrong way. An Embedding layer is just a trainable look-up table: you give it an integer , which is the index of the word in the vocabulary, and it returns the word-vector (ie word embedding) of the given index. Therefore, its input must be the indices of the words in a sentence.

As an example, if the indices of the words "over", "the", "to" and "find" are 43, 6, 9 and 33 respectively, then the input of the Embedding layer would be an array of those indices, ie [43, 6, 9, 33] . Therefore, the training data must have a shape of (num_samples, num_words_in_a_sentence) . In your case, it would be (100, 4) . In other words, you don't need to one-hot encode the words for the input data. You can also use word indices as the labels as well if you use sparse_categorical_crossentropy as the loss function instead.