TensorFlow running out of memory (ResourceExhaustedError) when executing embedding_lookup

Question

I am using a pre-tained vector to create an embedding like so

import numpy
import gensim
import tensorflow
ft_model=gensim.models.KeyedVectors.load_word2vec_format("ft_model.vec")
vocabulary=ft_model.vocab
embeddings=numpy.array([ft_model.word_vec(x) for x in vocabulary.keys()])

vocabulary_size=len(vocabulary)
embedding_size=embeddings.shape[1]

W=tensorflow.Variable(
    tensorflow.constant(0.0, shape=[vocabulary_size, embedding_size]),
    trainable=False,
    name="W"
)
embedding_placeholder=tensorflow.placeholder(
    tensorflow.float32,[vocabulary_size,embedding_size],
    name="fasttext_vector"
)
embedding_init=W.assign(embedding_placeholder)
data_placeholder=tensorflow.placeholder(tensorflow.int32,shape=[None, max_length])
embedding_layer=tensorflow.nn.embedding_lookup(W, data_placeholder)

I get an error after it briefly runs through 1 or two training batches and the code crashes completely!

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[5000,14621,100]

The stack trace cleary states that this is caused by the embedding_layer=tensorflow.nn.embedding_lookup(W, data_placeholder) line. Any idea what could be causing this? 100 is the embedding size but those other numbers (5000, 14621) are rather strange, larger than I exected, and seem to be causing TensorFlow to completely chew up all GPU memory! embedding lookups seem like such a common thing and the .vec file I am incorporating is very small.

Answer 1

It could be that you computer runs out of memory (RAM). Take a look at the taskmanager before you initiate your model. I have 16 GB and was at 79%, so it ran out. It might help using a jupyter notebook to see the amount of Ram left after having the data prepared