LSTM predicting continuous values

Question

Hello I am new to LSTM models and I want some help. I have a dataset where I have some words in my X vector and I want to use LSTM embedding layer to predict my y vector which consists with values between 0-1. I have seen a lot of examples where y vector consists of classes and we get a classification problem but here in this regression problem I cant find the solution.
I use this code below. First I import some libraries and then i try to use a simple LSTM deep network to predict y. Moreover as you see I use keras embedding layer to predict the y value

import numpy as np
from tensorflow.keras.callbacks import EarlyStopping
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.layers import Input, Embedding, LSTM, SpatialDropout1D, Dense, Dropout
from tensorflow.keras.models import Model, Sequential```
you can see the X and y vectors

X_train

array(['sinus surgery yesterday recover binging game throne start beginning hope not long mend',
       'respect game throne good fight scene week barry like far generally amazing episode television bill hader direction win Emmi offend wanton consumerism game throne oreos going eat nonstop remove face earth face end humanity human game throne wait night battle actually lie plan realistic far type game throne viewer happen happen end big winner game throne viewer sibling write article night game throne episode social medium absolute core game throne experience point watch thing time woman quickly accumulate power rule man mistake dead uncontrollable kid breathe fire game throne realistic family nerd weirdo end run world game throne basically story silicon valley game throne completely wipe trump trend story list winter feel good english major let offer insight subtle foreshadowing game throne threat winter come battle winterfell stoner guide game throne minute series end start episode feel like new weekend family role play game throne ramsey wife cersei kid joffrey finally way people internet impossible open page scroll timeline game throne spoiler minute east coast release finally kick online habit come popular news list game throne subject rise petty politic trump tantrum remain air good nation winner game throne officially',
       'wow people sign petition remake game throne season competent writer petition',
       ...,
       'star war rian johnson game throne creator work new trilogy honestly not mind bit think work fantastic new trilogy',
       'watch game throne documentary tonight',
       'game throne trash glad tell people read book'], dtype=object)

y_train

array([0.23024851, 0.47096001, 0.2237812 , ..., 0.07742384, 0.05732593,
       0.06153279])```

tokenizer = Tokenizer()
tokenizer.fit_on_texts(X_train)

max_length = max([len(s) for s in X_train])
vocab_size = len(tokenizer.word_index) + 1
X_train_tokens = tokenizer.texts_to_sequences(X_train)
X_train_pad = pad_sequences(X_train_tokens, 
                maxlen= max_length, padding= 'post')
embedding_dim = 100

model = Sequential()
model.add(Embedding(vocab_size, embedding_dim, input_length = max_length))
model.add(SpatialDropout1D(0.25))
model.add(LSTM(80, dropout=0.5))
model.add(Dropout(0.2))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='mse',optimizer='rmsprop')
print(model.summary())

model.fit(  X_train_pad, y_train, validation_split=0.2, batch_size= 64, epochs= 20)


Epoch 1/20
5031/5031 [==============================] - 612s 120ms/step - loss: 0.0062 - val_loss: 0.0055
Epoch 2/20
5031/5031 [==============================] - 603s 120ms/step - loss: 0.0056 - val_loss: 0.0057
Epoch 3/20
5031/5031 [==============================] - 601s 119ms/step - loss: 0.0056 - val_loss: 0.0060
Epoch 4/20
5031/5031 [==============================] - 603s 120ms/step - loss: 0.0056 - val_loss: 0.0055
Epoch 5/20
5031/5031 [==============================] - 601s 119ms/step - loss: 0.0056 - val_loss: 0.0057
Epoch 6/20
5031/5031 [==============================] - 602s 120ms/step - loss: 0.0055 - val_loss: 0.0060
Epoch 7/20
5031/5031 [==============================] - 600s 119ms/step - loss: 0.0056 - val_loss: 0.0060
Epoch 8/20
5031/5031 [==============================] - 603s 120ms/step - loss: 0.0056 - val_loss: 0.0058
Epoch 9/20
5031/5031 [==============================] - 601s 120ms/step - loss: 0.0056 - val_loss: 0.0055
Epoch 10/20
5031/5031 [==============================] - 607s 121ms/step - loss: 0.0055 - val_loss: 0.0057
Epoch 11/20
5031/5031 [==============================] - 604s 120ms/step - loss: 0.0056 - val_loss: 0.0056
Epoch 12/20
5031/5031 [==============================] - 600s 119ms/step - loss: 0.0055 - val_loss: 0.0059
Epoch 13/20
5031/5031 [==============================] - 600s 119ms/step - loss: 0.0056 - val_loss: 0.0059
Epoch 14/20
5031/5031 [==============================] - 605s 120ms/step - loss: 0.0056 - val_loss: 0.0062
Epoch 15/20
5031/5031 [==============================] - 605s 120ms/step - loss: 0.0056 - val_loss: 0.0055
Epoch 16/20
5031/5031 [==============================] - 602s 120ms/step - loss: 0.0055 - val_loss: 0.0056
Epoch 17/20
5031/5031 [==============================] - 604s 120ms/step - loss: 0.0055 - val_loss: 0.0059
Epoch 18/20
5031/5031 [==============================] - 601s 119ms/step - loss: 0.0054 - val_loss: 0.0058
Epoch 19/20
5031/5031 [==============================] - 599s 119ms/step - loss: 0.0056 - val_loss: 0.0057
Epoch 20/20
5031/5031 [==============================] - 599s 119ms/step - loss: 0.0056 - val_loss: 0.0054

As you can see the validation loss is extremely small and when I try to predict some values I the same value as prediction. How can I fix it? Is it possible to predict a continuous vector with word embedding or you can predict only classes?

Answer 1

I have read the entire code you display and can point out that you missed a line here:

model.add(Dense(1, activation='sigmoid')).

Here we need activation = 'linear' since you are doing regression, or leave the activation parameter not defined, since by default is linear.