Keras - Autoencoder for Text Analysis

Question

So I'm trying to create an autoencoder that will take text reviews and find a lower dimensional representation. I'm using keras and I want my loss function to compare the output of the AE to the output of the embedding layer. Unfortunately, it gives me the following error. I'm pretty sure the problem is with my loss function but I can't seem to resolve the issue.

Autoencoder

print X_train.shape
input_i = Input(shape=(200,))
embedding = Embedding(input_dim=weights.shape[0],output_dim=weights.shape[1],
                      weights=[weights])(input_i)
encoded_h1 = Dense(64, activation='tanh')(embedding)
encoded_h2 = Dense(32, activation='tanh')(encoded_h1)
encoded_h3 = Dense(16, activation='tanh')(encoded_h2)
encoded_h4 = Dense(8, activation='tanh')(encoded_h3)
encoded_h5 = Dense(4, activation='tanh')(encoded_h4)
latent = Dense(2, activation='tanh')(encoded_h5)
decoder_h1 = Dense(4, activation='tanh')(latent)
decoder_h2 = Dense(8, activation='tanh')(decoder_h1)
decoder_h3 = Dense(16, activation='tanh')(decoder_h2)
decoder_h4 = Dense(32, activation='tanh')(decoder_h3)
decoder_h5 = Dense(64, activation='tanh')(decoder_h4)

output = Dense(weights.shape[1], activation='tanh')(decoder_h5)

autoencoder = Model(input_i,output)
encoder = Model(input_i,latent)

print autoencoder.summary()

import keras.backend as K
import tensorflow as tf
def embedded_mse(x_true, e_pred):
    print output
    print embedding
    mse = K.mean(K.square(output - embedding))
    print mse

    return tf.Session().run(mse)
autoencoder.compile(optimizer='adadelta',
                    loss=embedded_mse)
autoencoder.fit(X_train,X_train,epochs=10,
                batch_size=256, validation_split=.1)

Output

(100000, 200)
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_47 (InputLayer)        (None, 200)               0         
_________________________________________________________________
embedding_31 (Embedding)     (None, 200, 100)          21833700  
_________________________________________________________________
dense_528 (Dense)            (None, 200, 64)           6464      
_________________________________________________________________
dense_529 (Dense)            (None, 200, 32)           2080      
_________________________________________________________________
dense_530 (Dense)            (None, 200, 16)           528       
_________________________________________________________________
dense_531 (Dense)            (None, 200, 8)            136       
_________________________________________________________________
dense_532 (Dense)            (None, 200, 4)            36        
_________________________________________________________________
dense_533 (Dense)            (None, 200, 2)            10        
_________________________________________________________________
dense_534 (Dense)            (None, 200, 4)            12        
_________________________________________________________________
dense_535 (Dense)            (None, 200, 8)            40        
_________________________________________________________________
dense_536 (Dense)            (None, 200, 16)           144       
_________________________________________________________________
dense_537 (Dense)            (None, 200, 32)           544       
_________________________________________________________________
dense_538 (Dense)            (None, 200, 64)           2112      
_________________________________________________________________
dense_539 (Dense)            (None, 200, 100)          6500      
=================================================================
Total params: 21,852,306
Trainable params: 21,852,306
Non-trainable params: 0
_________________________________________________________________
None
Tensor("dense_539/Tanh:0", shape=(?, 200, 100), dtype=float32)
Tensor("embedding_31/Gather:0", shape=(?, 200, 100), dtype=float32)
Tensor("loss_48/dense_539_loss/Mean:0", shape=(), dtype=float32)

Error

---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
<ipython-input-155-a18e0c32f59b> in <module>()
      1 autoencoder.compile(optimizer='adadelta',
----> 2                     loss=embedded_mse)
      3 autoencoder.fit(X_train,embedding,epochs=10,
      4                 batch_size=256, validation_split=.1)

/home/andrew/.local/lib/python2.7/site-packages/keras/engine/training.pyc in compile(self, optimizer, loss, metrics, loss_weights, sample_weight_mode, weighted_metrics, target_tensors, **kwargs)
    848                 with K.name_scope(self.output_names[i] + '_loss'):
    849                     output_loss = weighted_loss(y_true, y_pred,
--> 850                                                 sample_weight, mask)
    851                 if len(self.outputs) > 1:
    852                     self.metrics_tensors.append(output_loss)

/home/andrew/.local/lib/python2.7/site-packages/keras/engine/training.pyc in weighted(y_true, y_pred, weights, mask)
    448         """
    449         # score_array has ndim >= 2
--> 450         score_array = fn(y_true, y_pred)
    451         if mask is not None:
    452             # Cast the mask to floatX to avoid float64 upcasting in theano

<ipython-input-153-73211fc383a5> in embedded_mse(x_true, e_pred)
      7     print mse
      8 
----> 9     return tf.Session().run(mse)

/home/andrew/.local/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in run(self, fetches, feed_dict, options, run_metadata)
    893     try:
    894       result = self._run(None, fetches, feed_dict, options_ptr,
--> 895                          run_metadata_ptr)
    896       if run_metadata:
    897         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/home/andrew/.local/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _run(self, handle, fetches, feed_dict, options, run_metadata)
   1122     if final_fetches or final_targets or (handle and feed_dict_tensor):
   1123       results = self._do_run(handle, final_targets, final_fetches,
-> 1124                              feed_dict_tensor, options, run_metadata)
   1125     else:
   1126       results = []

/home/andrew/.local/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
   1319     if handle is None:
   1320       return self._do_call(_run_fn, self._session, feeds, fetches, targets,
-> 1321                            options, run_metadata)
   1322     else:
   1323       return self._do_call(_prun_fn, self._session, handle, feeds, fetches)

/home/andrew/.local/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _do_call(self, fn, *args)
   1338         except KeyError:
   1339           pass
-> 1340       raise type(e)(node_def, op, message)
   1341 
   1342   def _extend_graph(self):

InvalidArgumentError: You must feed a value for placeholder tensor 'input_47' with dtype float and shape [?,200]
     [[Node: input_47 = Placeholder[dtype=DT_FLOAT, shape=[?,200], _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

Caused by op u'input_47', defined at:
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/andrew/.local/lib/python2.7/site-packages/ipykernel_launcher.py", line 16, in <module>
    app.launch_new_instance()
  File "/home/andrew/.local/lib/python2.7/site-packages/traitlets/config/application.py", line 658, in launch_instance
    app.start()
  File "/home/andrew/.local/lib/python2.7/site-packages/ipykernel/kernelapp.py", line 477, in start
    ioloop.IOLoop.instance().start()
  File "/home/andrew/.local/lib/python2.7/site-packages/zmq/eventloop/ioloop.py", line 177, in start
    super(ZMQIOLoop, self).start()
  File "/home/andrew/.local/lib/python2.7/site-packages/tornado/ioloop.py", line 888, in start
    handler_func(fd_obj, events)
  File "/home/andrew/.local/lib/python2.7/site-packages/tornado/stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "/home/andrew/.local/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 440, in _handle_events
    self._handle_recv()
  File "/home/andrew/.local/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 472, in _handle_recv
    self._run_callback(callback, msg)
  File "/home/andrew/.local/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 414, in _run_callback
    callback(*args, **kwargs)
  File "/home/andrew/.local/lib/python2.7/site-packages/tornado/stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "/home/andrew/.local/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 283, in dispatcher
    return self.dispatch_shell(stream, msg)
  File "/home/andrew/.local/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 235, in dispatch_shell
    handler(stream, idents, msg)
  File "/home/andrew/.local/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 399, in execute_request
    user_expressions, allow_stdin)
  File "/home/andrew/.local/lib/python2.7/site-packages/ipykernel/ipkernel.py", line 196, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)
  File "/home/andrew/.local/lib/python2.7/site-packages/ipykernel/zmqshell.py", line 533, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "/home/andrew/.local/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2718, in run_cell
    interactivity=interactivity, compiler=compiler, result=result)
  File "/home/andrew/.local/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2822, in run_ast_nodes
    if self.run_code(code, result):
  File "/home/andrew/.local/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2882, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-152-7732fda181fc>", line 2, in <module>
    input_i = Input(shape=(200,))
  File "/home/andrew/.local/lib/python2.7/site-packages/keras/engine/topology.py", line 1436, in Input
    input_tensor=tensor)
  File "/home/andrew/.local/lib/python2.7/site-packages/keras/legacy/interfaces.py", line 87, in wrapper
    return func(*args, **kwargs)
  File "/home/andrew/.local/lib/python2.7/site-packages/keras/engine/topology.py", line 1347, in __init__
    name=self.name)
  File "/home/andrew/.local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 442, in placeholder
    x = tf.placeholder(dtype, shape=shape, name=name)
  File "/home/andrew/.local/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 1548, in placeholder
    return gen_array_ops._placeholder(dtype=dtype, shape=shape, name=name)
  File "/home/andrew/.local/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 2094, in _placeholder
    name=name)
  File "/home/andrew/.local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
    op_def=op_def)
  File "/home/andrew/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/andrew/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1204, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'input_47' with dtype float and shape [?,200]
     [[Node: input_47 = Placeholder[dtype=DT_FLOAT, shape=[?,200], _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

Answer 1

There are some issues with your question (eg what is weights , used in the Embedding & final Dense layers arguments?). Still, I think a simpler approach would be to disentangle the embedding and the autoencoding parts (they are independent), by building first a simple embedding model and then use its outputs (with predict ) to feed your autoencoder. This way you don't have to define a custom loss (BTW, print statements in such functions are not a good idea).

Without knowing the details of your data, the following 2 models compile OK:

Embedding model (quick adaptation from the docs )

model = Sequential()
model.add(Embedding(1000, 64))
model.compile('rmsprop', 'mse')

Autoencoder:

input_i = Input(shape=(200,100))
encoded_h1 = Dense(64, activation='tanh')(input_i)
encoded_h2 = Dense(32, activation='tanh')(encoded_h1)
encoded_h3 = Dense(16, activation='tanh')(encoded_h2)
encoded_h4 = Dense(8, activation='tanh')(encoded_h3)
encoded_h5 = Dense(4, activation='tanh')(encoded_h4)
latent = Dense(2, activation='tanh')(encoded_h5)
decoder_h1 = Dense(4, activation='tanh')(latent)
decoder_h2 = Dense(8, activation='tanh')(decoder_h1)
decoder_h3 = Dense(16, activation='tanh')(decoder_h2)
decoder_h4 = Dense(32, activation='tanh')(decoder_h3)
decoder_h5 = Dense(64, activation='tanh')(decoder_h4)

output = Dense(100, activation='tanh')(decoder_h5)

autoencoder = Model(input_i,output)

autoencoder.compile('adadelta','mse')

After adapting the above models parameters to your case, this should work fine:

X_embedded = model.predict(X_train)
autoencoder.fit(X_embedded,X_embedded,epochs=10,
            batch_size=256, validation_split=.1)