Tensorflow embedding running out of space

Question

I am trying to create an embedding for 1,000,000 words on tensorflow. Each word will have a 256 float32 vector representing the word. The issue is that I keep running out of memory. This does not make sence to me since I have 8GB of memory on my GTX 1080. The embedding should only take up 1e6 * 256 * 4 = 1 Gb of memory. I also have another matrix on the output which is of the same size. Other than that there are a few other tensors which should be small in comparison. Therefore I only see about 2 - 3 GB worth of memory needed to store the model and it is failing when I call sess.run(tf.initialize_all_variables()) . Where is all my memory going and do you have any advice for how I can get around this?

import tensorflow as tf
import nltk
import numpy as np
import os
import multiprocessing
import itertools
import pickle
from unidecode import unidecode

BATCH_SIZE = 32
TIME_STEPS = 64
WORD_VEC_SIZE = 256

words, training_data = pickle.load(open('vocab.pickle', 'rb'))
word2index = {w:i for i, w in enumerate(words)}
index2word = {i:w for i, w in enumerate(words)}

input_tensor = tf.placeholder(tf.int32, (BATCH_SIZE, TIME_STEPS + 1), 'input_tensor')
embedding = tf.Variable(tf.random_uniform((len(words), WORD_VEC_SIZE), -1, 1), name = 'embedding')

rnn = tf.nn.rnn_cell.BasicRNNCell(WORD_VEC_SIZE)
state = tf.zeros((BATCH_SIZE, rnn.state_size))
input_vectors = tf.nn.embedding_lookup([embedding], input_tensor[:, :TIME_STEPS])
cost = 0

with tf.variable_scope('rnn') as scope:
    W_out = tf.get_variable('W_out', (WORD_VEC_SIZE, len(words)), initializer = tf.truncated_normal_initializer(0.0, 1 / np.sqrt(WORD_VEC_SIZE)))
    b_out = tf.get_variable('b_out', (len(words), ), initializer = tf.truncated_normal_initializer(0.0, 0.01))
    for t in range(TIME_STEPS):
        y, state = rnn(tf.reshape(input_vectors[:, t, :], (-1, WORD_VEC_SIZE)), state)
        cost += tf.reduce_mean(tf.nn.sampled_softmax_loss(W_out, b_out, y, tf.reshape(input_tensor[:, t + 1], (-1, 1)), 1000, len(words)))
        scope.reuse_variables()

train_step = tf.train.AdamOptimizer(1e-4).minimize(cost)

sess = tf.Session()
sess.run(tf.initialize_all_variables())
saver = tf.train.Saver()

Answer 1

What I was not accounting for was the AdamOptimizer. I forgot that this needs to store various parameters for each of the weights in my model. When I changed to a GraidentDecent optimizer it now fits on my GPU.