在keras模型中使用预先训练的单词嵌入？

Question

I was following this github code from keras team on how to use pre-trained word embeddings. I was able to understand most of it but I've a doubt regarding vector sizes. I was hoping someone could help me out.

First we define Tokenizer(num_words=MAX_NUM_WORDS)

Accoding to keras docs for Tokenizer() num_words argument only consider MAX_NUM_WORDS - 1 so if MAX_NUM_WORDS=20000 I'll have around 19999 words.

num_words : the maximum number of words to keep, based on word frequency. Only the most common num_words-1 words will be kept.

Next in the code we prepare a Embedding Matrix based on glove vectors. When doing that we are considering an matrix of size (20001, 100) np.zeros((MAX_NUM_WORDS+1, 100)) . I couldn't get why we are consider a matrix of 20001 if we have only 19999 words in our vocabulary.

Also then we are passing num_words to the Embedding layer. According to Embedding layer docs for input_dim argument, It says,

input_dim : int > 0. Size of the vocabulary, ie maximum integer index + 1.

embedding_layer = Embedding(input_dim=num_words,
                            output_dim=EMBEDDING_DIM,
                            embeddings_initializer=Constant(embedding_matrix),
                            input_length=MAX_SEQUENCE_LENGTH,
trainable=False)

Here our vocabulary size will be 19999 according to Tokenizer() function right? So why are we passing 20001 as input_dim

Here's a small snippet of the code taken from that github link.

MAX_NUM_WORDS = 20000
MAX_SEQUENCE_LENGTH = 1000
EMBEDDING_DIR = 100

tokenizer = Tokenizer(num_words=MAX_NUM_WORDS)
tokenizer.fit_on_texts(texts)
sequences = tokenizer.texts_to_sequences(texts)

data = pad_sequences(sequences, maxlen=MAX_SEQUENCE_LENGTH)

# prepare embedding matrix
num_words = MAX_NUM_WORDS + 1
embedding_matrix = np.zeros((num_words, EMBEDDING_DIM))
for word, i in word_index.items():
    if i > MAX_NUM_WORDS:
        continue
    embedding_vector = embeddings_index.get(word)
    if embedding_vector is not None:
        embedding_matrix[i] = embedding_vector

embedding_layer = Embedding(num_words,
                            EMBEDDING_DIM,
                            embeddings_initializer=Constant(embedding_matrix),
                            input_length=MAX_SEQUENCE_LENGTH,
                            trainable=False)

Answer 1

For the embedding, input dim (num_words in the below code) is the size of the vocabulary. For example, if your data is integer encoded to values between 0-10, then the size of the vocabulary would be 11 words. That is the reason 1 is added to the min of len(word_index) and MAX_NUM_WORDS.

Embedding matrix will have the the dimension of vocabulary size and vector length

embedding_layer = Embedding(num_words,
                            EMBEDDING_DIM,
                            embeddings_initializer=Constant(embedding_matrix),
                            input_length=MAX_SEQUENCE_LENGTH,
                            trainable=False)

num_words = min(MAX_NUM_WORDS, len(word_index)) + 1

Have created a simple tokenizer to explain this.

t  = Tokenizer(num_words=5)
fit_text = ["The earth is an awesome place live"]
t.fit_on_texts(fit_text)
word_index = t.word_index
​
print('word_index : ',word_index)
print('len word_index : ',len(t.word_index))
word_index :  {'the': 1, 'earth': 2, 'is': 3, 'an': 4, 'awesome': 5, 'place': 6, 'live': 7}
len word_index :  7

In the below case, you are covering a vocabulary of size 4 only because tokenizer indexing starts from 1.

embedding_matrix = np.zeros((5, 10))
embedding_matrix
array([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])

for word, i in word_index.items():
    if i < 5:       
        embedding_matrix[i] = [0,1,0,0,0,0,0,0,0,0]

print (embedding_matrix)
[[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]]

In the below case, you need to add 1 (5+1) to cover the vocabulary of size 5 to cover for the index 0

embedding_matrix = np.zeros((6, 10))
for word, i in word_index.items():
    if i < 6:       
        embedding_matrix[i] = [0,1,0,0,0,0,0,0,0,0]

print (embedding_matrix)

[[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]]

Answer 2

I think your doubt is valid. The change was made in this commit of the code to keep the word with index = MAX_NUM_WORDS . Before that there was a commit on Tokenizer to make it keep num_words words instead of num_words - 1 words. But this change of Tokenizer was reverted afterwards. So I guess the author of the example update might have assumed that Tokenizer kept num_words words when the update was committed.