LSTM keras 多个特性：我做错了什么？

Question

The code below predicts Close value (stock prices) with 3 inputs: Close , Open and Volume . Dataset:

             Close    Open   Volume
Date                               
2019-09-20  5489.0  5389.0  1578781
2019-09-23  5420.0  5460.0   622325
2019-09-24  5337.5  5424.0   688395
2019-09-25  5343.5  5326.5   628849
2019-09-26  5387.5  5345.0   619344
...            ...     ...      ...
2020-03-30  4459.0  4355.0  1725236
2020-03-31  4715.0  4550.0  2433310
2020-04-01  4674.5  4596.0  1919728
2020-04-02  5050.0  4865.0  3860103
2020-04-03  5204.5  5050.0  3133078

[134 rows x 3 columns]

Info:

<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 134 entries, 2019-09-20 to 2020-04-03
Data columns (total 3 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   Close   134 non-null    float64
 1   Open    134 non-null    float64
 2   Volume  134 non-null    int64  
dtypes: float64(2), int64(1)

The question is how to correct script to get right prediction with 3 features last 10 days, because I get this:

Epoch 1/1
64/64 [==============================] - 6s 88ms/step - loss: 37135470.9219
[[32.588608]
 [32.587284]
 [32.586754]
 [32.587196]
 [32.58649 ]
 [32.58663 ]
 [32.586098]
 [32.58682 ]
 [32.586452]
 [32.588108]]
rmse: 4625.457010985681

The problem remains even if I remove scaling ( fit_transform ) at all. In other topics I was told there is no need to scale y_train . Full script code:

from math import sqrt
from numpy import concatenate
import pandas as pd
from sklearn.preprocessing import MinMaxScaler
from sklearn.preprocessing import LabelEncoder
from sklearn.metrics import mean_squared_error
from keras.models import Sequential
from keras.layers import Dense, Dropout, Embedding
from keras.layers import LSTM
import numpy as np
from datetime import datetime, timedelta
import yfinance as yf

start = (datetime.now() - timedelta(days=200)).strftime("%Y-%m-%d")
end = (datetime.now() - timedelta(days=1)).strftime("%Y-%m-%d")
df = yf.download(tickers="LKOH.ME", start=start, end=end, interval="1d")
dataset = df.loc[start:end].filter(['Close', 'Open', 'Volume']).values
scaler = MinMaxScaler(feature_range=(0,1))

training_data_len = len(dataset) - 10 # last 10 days to test
train_data = dataset[0:int(training_data_len), :]
x_train = []
y_train = []

for i in range(60, len(train_data)):
    x_train.append(train_data[i-60:i, :]) # get all 3 features
    y_train.append(train_data[i, 0]) # 0 means we predict Close

x_train, y_train = np.array(x_train), np.array(y_train)
x_train = x_train.reshape((x_train.shape[0], x_train.shape[1]*x_train.shape[2])) # convert to 2d for fit_transform()
x_train = scaler.fit_transform(x_train)
x_train = np.reshape(x_train, (x_train.shape[0], x_train.shape[1], 1))

model = Sequential()
# Do I need to change it to input_shape=(x_train.shape[1], 3), because of 3 features?
model.add(LSTM(50, return_sequences=True, input_shape=(x_train.shape[1], 1)))
model.add(LSTM(50))
model.add(Dense(25))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mean_squared_error')
model.fit(x_train, y_train, batch_size=1, epochs=1)

test_data = dataset[training_data_len - 60:, :]
x_test = []
y_test = dataset[training_data_len:, 0]
for i in range(60, len(test_data)):
    x_test.append(test_data[i-60:i, :])

x_test = np.array(x_test)
x_test = x_test.reshape((x_test.shape[0], x_test.shape[1]*x_test.shape[2]))
x_test = scaler.fit_transform(x_test)
x_test = np.reshape(x_test, (x_test.shape[0], x_test.shape[1], 1))

predictions = model.predict(x_test)
print(predictions)
print('rmse:', np.sqrt(np.mean(((predictions - y_test) ** 2))))

Answer 1

As @rvinas has already mentioned, we need to scale the values and then use inverse_transform to get the desired predicted outcome. You can find the reference here .

After making some small changes in the code and I was able to come up with satisfactory results. We can play around with the data scaling methodologies and model architectures to improve the results.

After some enhancements

from math import sqrt
from numpy import concatenate
import pandas as pd
from sklearn.preprocessing import MinMaxScaler
from sklearn.preprocessing import LabelEncoder
from sklearn.metrics import mean_squared_error
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, Embedding
from tensorflow.keras.layers import LSTM
from tensorflow.keras.optimizers import SGD
import numpy as np
from datetime import datetime, timedelta
import yfinance as yf

start = (datetime.now() - timedelta(days=200)).strftime("%Y-%m-%d")
end = (datetime.now() - timedelta(days=1)).strftime("%Y-%m-%d")
df = yf.download(tickers="LKOH.ME", start=start, end=end, interval="1d")
dataset = df.loc[start:end].filter(['Close', 'Open', 'Volume']).values
scaler = MinMaxScaler(feature_range=(0,1))


dataset = scaler.fit_transform(dataset)
training_data_len = len(dataset) - 10 # last 10 days to test
train_data = dataset[0:int(training_data_len), :]
x_train = []
y_train = []

for i in range(60, len(train_data)):
    x_train.append(train_data[i-60:i, :]) # get all 3 features
    y_train.append(train_data[i, 0]) # 0 means we predict Close


x_train, y_train = np.array(x_train), np.array(y_train)
x_train = x_train.reshape((x_train.shape[0], x_train.shape[1]*x_train.shape[2])) # convert to 2d for fit_transform()
x_train_scale = scaler.fit(x_train)
x_train = np.reshape(x_train, (x_train.shape[0], x_train.shape[1], 1))

model = Sequential()
# Do I need to change it to input_shape=(x_train.shape[1], 3), because of 3 features?
# yes, i did that.

model.add(LSTM(units=50,return_sequences=True, kernel_initializer='random_uniform', input_shape=(x_train.shape[1], 1)))
model.add(Dropout(0.2))
model.add(LSTM(units=50,return_sequences=True, kernel_initializer='random_uniform'))
model.add(Dropout(0.2))
model.add(LSTM(units=50,return_sequences=True, kernel_initializer='random_uniform'))
model.add(Dropout(0.2))
model.add(LSTM(units=50, kernel_initializer='random_uniform'))
model.add(Dropout(0.2))
model.add(Dense(units=25, activation='relu'))
model.add(Dense(units=1))

# compile model
model.compile(optimizer='adam', loss='mean_squared_error')
model.summary()
model.fit(x_train, y_train, batch_size=5, epochs=2)

test_data = dataset[training_data_len - 60:, :]
x_test = []
y_test = dataset[training_data_len:, 0]
for i in range(60, len(test_data)):
    x_test.append(test_data[i-60:i, :])

x_test = np.array(x_test)
x_test = x_test.reshape((x_test.shape[0], x_test.shape[1]*x_test.shape[2]))
x_test = np.reshape(x_test, (x_test.shape[0], x_test.shape[1], 1))

predictions = model.predict(x_test)
# predictions = y_train_scale.inverse_transform(predictions)
print(predictions)
print('rmse:', np.sqrt(np.mean(((predictions - y_test) ** 2))))

Predictions 1:

start = (datetime.now() - timedelta(days=200)).strftime("%Y-%m-%d")

opt = SGD(lr=0.01, momentum=0.9, clipnorm=1.0, clipvalue=0.5)
model.compile(loss='mean_squared_error', optimizer=opt)

[[0.6151125 ]
 [0.6151124 ]
 [0.6151121 ]
 [0.6151119 ]
 [0.61511195]
 [0.61511236]
 [0.61511326]
 [0.615114  ]
 [0.61511385]
 [0.6151132 ]]
rmse: 0.24450220836260966

Prediction 2:

start = (datetime.now() - timedelta(days=1000)).strftime("%Y-%m-%d")

model.compile(optimizer='adam', loss='mean_squared_error')

[[0.647125  ]
 [0.6458076 ]
 [0.6405072 ]
 [0.63450944]
 [0.6315386 ]
 [0.6384401 ]
 [0.65666   ]
 [0.68073314]
 [0.703547  ]
 [0.72095114]]
rmse: 0.1236932687978488

Stock market prices are highly unpredictable and volatile. This means that there are no consistent patterns in the data that allow you to model stock prices over time near-perfectly. So this needs a lot of R&D to come up with a good strategy.

Things that you can do:

1. Add more training data to your model, so that it is able to generalize it better.
2. Make the model deeper. Play around with the model Hyperparameters to squeeze out the performance of the model. You can find a good reference here about hyperparameter tuning .

You can find more information about various other data preprocessing techniques and model architecture from reference links below:

Stock Market Predictions with LSTM in Python

Machine Learning to Predict Stock Prices

Answer 2

Even though the posted answer is technically correct and provides useful references, I found it a bit annoying that the results of fitting do not make much sense (you can notice that predictions are constant, even though the y_test isn't). Yes, scaling fixes the loss - with the values in the order of 1000 the L2 measure makes any gradient-based algorithm very unstable and Rishab's answer addresses that. Here is my code snippet. With the following changes in addition to the scaling:

• Use more data. I randomly chose 10000 days, but if there is more, you probably will get better results. 200 points are not sufficient to get any convergence better than the straight line.
• With more points use a larger batch as otherwise, it'll take a while to fit
• With the larger batch, use more epochs (although in this case, more than 3 do not produce any better convergence)

Lastly, do not just look at the RMSE, plot your data. Small RMSE doesn't necessarily mean there is any meaningful fit.

With the snippet below I've got a somewhat good fit on the train data. And foreseeing the questions: yes, I totally know that I'm overfitting the data, but that is what the convergence should be doing at the very least here as fitting a straight line is much less meaningful for this of problem. This, at least, pretends to predict something.

from math import sqrt
from numpy import concatenate
import pandas as pd
from sklearn.preprocessing import MinMaxScaler
from sklearn.preprocessing import LabelEncoder
from sklearn.metrics import mean_squared_error
from keras.models import Sequential
from keras.layers import Dense, Dropout, Embedding
from keras.layers import LSTM
import numpy as np
from datetime import datetime, timedelta
import yfinance as yf
from matplotlib import pyplot as plt

start = (datetime.now() - timedelta(days=10000)).strftime("%Y-%m-%d")
end = (datetime.now() - timedelta(days=1)).strftime("%Y-%m-%d")
df = yf.download(tickers="LKOH.ME", start=start, end=end, interval="1d")
scaler = MinMaxScaler(feature_range=(0,1))
dataset = scaler.fit_transform(df.loc[start:end].filter(['Close', 'Open', 'Volume']).values)

training_data_len = len(dataset) - 10 # last 10 days to test
train_data = dataset[0:int(training_data_len), :]
x_train = []
y_train = []

for i in range(60, len(train_data)):
    x_train.append(train_data[i-60:i, :]) # get all 3 features
    y_train.append(train_data[i, 0]) # 0 means we predict Close

x_train, y_train = np.array(x_train), np.array(y_train)
x_train = x_train.reshape((x_train.shape[0], x_train.shape[1]*x_train.shape[2])) # convert to 2d for fit_transform()
x_train = np.reshape(x_train, (x_train.shape[0], x_train.shape[1], 1))

model = Sequential()
# Do I need to change it to input_shape=(x_train.shape[1], 3), because of 3 features?
model.add(LSTM(50, return_sequences=True, input_shape=(x_train.shape[1], 1)))
model.add(LSTM(50))
model.add(Dense(25))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mean_squared_error')
model.fit(x_train, y_train, batch_size=100, epochs=3)

test_data = dataset[training_data_len - 60:, :]
x_test = []
y_test = dataset[training_data_len:, 0]
for i in range(60, len(test_data)):
    x_test.append(test_data[i-60:i, :])

x_test = np.array(x_test)
x_test = x_test.reshape((x_test.shape[0], x_test.shape[1]*x_test.shape[2]))
x_test = np.reshape(x_test, (x_test.shape[0], x_test.shape[1], 1))

predictions = model.predict(x_test)
print(predictions)
print('rmse:', np.sqrt(np.mean(((predictions - y_test) ** 2))))

Here is the output:

>>> print(predictions)
[[0.64643383]
 [0.63276255]
 [0.6288108 ]
 [0.6320714 ]
 [0.6572328 ]
 [0.6998471 ]
 [0.7333    ]
 [0.7492812 ]
 [0.7503019 ]
 [0.75124526]]
>>> print('rmse:', np.sqrt(np.mean(((predictions - y_test) ** 2))))
rmse: 0.0712241892828221

To plot use, for training data fit

plt.plot(model.predict(x_train))
plt.plot(y_train)
plt.show()    

and for test predictions

plt.plot(model.predict(x_test))
plt.plot(y_test)
plt.show()