Keras 中嵌入层的详细信息

Question

I want to understand the Embedding layer in Keras. Can anyone help? I cannot find any good reference.

1. Which algorithm is working? I mean based on which paper or method? I want details to implement it myself.
2. how the training data for embedding is created? It is 2 words 2 words or more neighbors...
3. How the model updates the weights? is it first embedding then model, or both update together?
4. How the parameters is calculated? For example i have this simple model:

model = Sequential()

model.add(Embedding(input_dim=15, output_dim=8, input_length=4))

model.add(Flatten())

model.add(Dense(1, activation='sigmoid'))

model.summary()

and this is the summary:

Layer (type)                 Output Shape              Param 

=================================================================
embedding (Embedding)        (None, 4, 8)              120       
_________________________________________________________________
flatten (Flatten)            (None, 32)                0         
_________________________________________________________________
dense (Dense)                (None, 1)                 33  

how the 120 is calculated? Thanks in advance!

Answer 1

In terms of the number of parameters (120), that is the number of weights that need to be stored for the embedding layer specified. If the embedding layer has an input dimension of 15 (15 possible input categories or words) and an output dimension of 8 (each category represented by an 8-dimensional array), then the total number of weights or parameters stored by the embedding layer is 15x8=120.

The embedding weights themselves can either be provided prior to training (using pretrained weights: https://keras.io/examples/nlp/pretrained_word_embeddings/ ) or the weights can be learned during the model training process. In the latter case the method of learning is the same as the rest of the network, presumably some form of gradient descent in an effort to minimize loss against the provided target. You can choose to train or freeze the weights using the 'trainable' keyword argument when initializing the embedding layer.