Error in reverse scaling outputs predicted by a LSTM RNN

Question

I used the LSTM model to predict the future open price of a stock. Here the data was preprocessed and the model was built and trained without any errors, and I used Standard Scaler to scale down the values in the DataFrame. But while retrieving the predictions from the model, when I used the scaler.reverse() method it gave the following error.

ValueError: non-broadcastable output operand with shape (59,1) doesn't match the broadcast shape (59,4)

The complete code is a too big jupyter notebook to directly show, so I have uploaded it in a git repository

Answer 1

This is because the model is predicting output with shape (59, 1). But your Scaler was fit on (251, 4) data frame. Either create a new scaler on the data frame of the shape of y values or change your model dense layer output to 4 dimensions instead of 1. The data shape on which scaler is fit, it will take that shape only during scaler.inverse_transform .

Old Code - Shape (n,1)

trainY.append(df_for_training_scaled[i + n_future - 1:i + n_future, 0])

Updated Code - Shape (n,4) - use all 4 outputs

trainY.append(df_for_training_scaled[i + n_future - 1:i + n_future,:])

Answer 2

Normally you'd be re-scaling independent variables (features) as differences in scale can affect model calculations, but the dependent variable that you're trying to predict is normally left untouched. There's usually no reason to re-scale the dependent variable and scaling it makes it extremely difficult to interpret results.

The first line of documentation of StandardScaler class even specifies as much:

Standardize features by removing the mean and scaling to unit variance

You can optionally also scale labels, but once again this is not normally required.

So what I'd do in your place is (assuming your original dataframe contains 3 independent variables and 1 target variable) is this:

X = some_df.iloc[:, :3].values
y = some_df.iloc[3].values

scaler = StandardScaler()
X = scaler.fit_transform(X)
# And then goes everything as usual

Now, when you go to predict values you simply need to transform your input with the scaler in the same way it's been done before.

The better way, though, would be to add to your model a Normalization layer as a pre-processing step. This way you just feed raw data into your estimator and it handles all the nitty-gritty for you. And, similarly, you won't need to normalize data when generating predictions, the model will do everything for you. You could add something like:

from tensorflow.keras.layers.experimental.preprocessing import Normalization
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Input, Dense
from tensorflow.keras import Model

# this is your default batch_size
BATCH_SIZE = 128

# Here's your raw (non-normalized) X data
X = some_df.iloc[:, :3].values

norm = Normalization()
norm.adapt(X)
preprocess = Sequential([
    Input(shape=(BATCH_SIZE, 3)),
    norm
])

# Now finally, when you build your actual model you add 
# pre-processing step in the beginning
inp = preprocess()
x = Dense(64)(input)
x = Dense(128)(x)
x = Dense(1)(x)  
model = Model(inputs=inp, outputs=x) 

Here the pre-process step is a part of the model itself so once you do that you can just feed it raw data without any additional transformations.

This is what it will do:

# Skipping the imports as they are the same as above + numpy

X = np.array([[1, 2, 3], [10, 20, 40], [100, 200, 400]])

norm = Normalization()
norm.adapt(X)
preprocess = Sequential([
    Input(shape=(3, 3)),
    norm
]) 

x_new = preprocess(X)
print(x_new)

Out: tf.Tensor(
[[-0.80538726 -0.80538726 -0.807901  ]
 [-0.60404044 -0.60404044 -0.6012719 ]
 [ 1.4094278   1.4094278   1.4091729 ]], shape=(3, 3), dtype=float32)