ValueError: Error when checking target: expected activation_10 to have 2 dimensions, but got array with shape (118, 50, 1)

Question

I am trying to use LSTM to predict stock prices and I ran into the following error

this is my code:

from keras.layers.core import Dense, Activation, Dropout
from keras.layers.recurrent import LSTM
from keras.models import Sequential
import lstm, time

X_train, X_test, Y_train, Y_test = lstm.load_data('tata.csv', 50, True)

#build model
model = Sequential()

mode.add(LSTM(
    input_dim=1,
    output_dim=50,
    return_sequences=True))
model.add(Dropout(0.2))

model.add(LSTM(
    100,
    return_sequences=False))
model.add(Dropout(0.2))

model.add(Dense(
    output_dim=1,))
model.add(Activation('linear'))

start = time.time()
model.compile(loss='mse', optimizer='rmsprop')
print("compilation time: ", time.time - start)

#train the model
model.fit(
    X_train,
    Y_train,
    batch_size=512,
    nb_epoch=1,
    validation_split=0.05)

#predicting the prices
predictions = lstm.predict_sequences_multiple(model, X_test, 50, 50)
lstm.plot_results_multiple(predictions, Y_test, 50)

and this is the error I ran into:

ValueError: Error when checking target: expected activation_10 to have 2 dimensions, but got array with shape (118, 50, 1)

this is the full image

I am not able to understand where the problem is. Please look at the image I linked to get a better understanding.

Answer 1

This error occurs when the model input shape and data passed shape are different.

Below is an example where I have used your code but used different input file(as I couldn't import lstm ) and did some preprocessing to scale the input using MinMaxScaler , split dataset into Xtrain and Ytrain and finally convert list to np.array .

I have added X_train = X_train[:,:,np.newaxis] in the code to accommodate the X_train shape with model. Without this line the model will throw ValueError: Error when checking input: expected lstm_13_input to have 3 dimensions, but got array with shape (1975, 50) .

Full Code -

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from keras.layers.core import Dense, Activation, Dropout
from keras.layers.recurrent import LSTM
from keras.models import Sequential
import time

url = 'https://raw.githubusercontent.com/mwitiderrick/stockprice/master/NSE-TATAGLOBAL.csv'
dataset_train = pd.read_csv(url)
training_set = dataset_train.iloc[:, 1:2].values

dataset_train.head()

from sklearn.preprocessing import MinMaxScaler
sc = MinMaxScaler(feature_range=(0,1))
training_set_scaled = sc.fit_transform(training_set)

X_train = []
y_train = []
for i in range(60, 2035):
  X_train.append(training_set_scaled[i-50:i, 0])
  y_train.append(training_set_scaled[i, 0])

X_train, Y_train = np.array(X_train), np.array(y_train)
X_train = X_train[:,:,np.newaxis]

#build model
model = Sequential()

model.add(LSTM(
    input_dim=1,
    output_dim=50,
    return_sequences=True))
model.add(Dropout(0.2))

model.add(LSTM(
    100,
    return_sequences=False))
model.add(Dropout(0.2))

model.add(Dense(
    output_dim=1,))
model.add(Activation('linear'))

#model.summary()

start = time.time()
model.compile(loss='mse', optimizer='rmsprop')
print("compilation time: ", time.time() - start)

#train the model
model.fit(
    X_train,
    Y_train,
    batch_size=512,
    nb_epoch=1,
    validation_split=0.05)

Output -

/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:34: UserWarning: The `input_dim` and `input_length` arguments in recurrent layers are deprecated. Use `input_shape` instead.
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:34: UserWarning: Update your `LSTM` call to the Keras 2 API: `LSTM(return_sequences=True, input_shape=(None, 1), units=50)`
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:43: UserWarning: Update your `Dense` call to the Keras 2 API: `Dense(units=1)`
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:58: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.
compilation time:  0.021114110946655273
Train on 1876 samples, validate on 99 samples
Epoch 1/1
1876/1876 [==============================] - 3s 1ms/step - loss: 0.0468 - val_loss: 0.0067
<keras.callbacks.callbacks.History at 0x7fa58e0efd30>

Hope this answers your question. Thank You.